Commit Graph

74 Commits

Author SHA1 Message Date
Abdó Roig-Maranges
0e4afa9132 Make GmailFolder sync GMail labels
When synclabels config flag is set to "yes" for the GMail repo,
offlineimap fetches the message labels along with the messages, and
embeds them into the body under the header X-Keywords (or whatever
'labelsheader' was set to), as a comma separated list.

It also adds an extra pass to savemessageto, that performs label
synchronization on existing messages from GMail to local, the same way
it is done with flags.

We also introduce GmailMaildir repository that adds functionality to
change message labels.  It keeps track of messages modification time,
so one can quickly detect when the labels may have changed.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-05-06 23:36:06 +04:00
Abdó Roig-Maranges
9319ae212b Restructured folder/IMAP code
In preparation for GMail label sync, we had split our some functionality
that will be needed further into their own functions.  This also permitted
the code to look more compact and concise.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-05-06 23:03:21 +04:00
Abdó Roig-Maranges
697ca8a229 Changed NotImplementedException to NotImplementedError
It seems NotImplementedException does not exist. It must be a relic
from old Python...

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-05-02 19:23:01 +04:00
Eygene Ryabinkin
6cbd2498ae Refactoring: make functions to be private if we can
Make external API of class/module to be smaller, explicitely mark
all internal functions.  Also annotate methods that are implemented
as the part of the parent class interface.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-04-30 19:35:56 +04:00
Stefan Huber
af2d6dc5e1 Fix: Undefined variable 'this' should be 'self'
Pointyhat-to: Eygene Ryabinkin
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-03-06 10:48:34 +04:00
Eygene Ryabinkin
829c9cf4be Introduce dynamic filtering for repository folders
Allow people who want folder filtering to depend on the external
conditions or to make it dynamic for other reasons, to do what
they want.

New repository configuration knob 'dynamic_folderfilter' was
introduced; it defaults to 'False' that matches historical behaviour.

GitHub: #73
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-03-02 16:37:41 +04:00
Eygene Ryabinkin
69765a3ef0 Properly call error() function from ui.UIBase
Second argument is exception traceback, not the message; without this
tracebacks like mentioned in
  http://permalink.gmane.org/gmane.mail.imap.offlineimap.general/5712
were happening when this exception handling block was hit.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-08-30 21:49:09 +04:00
X-Ryl669
3bc66c0858 Add support for alternative message date synchronisation
Global or per-repository option utime_from_message tells OfflineIMAP
to set file modification time of messages pushed from one repository
to another basing on the message's "Date" header.

This is useful if you are doing some processing/finding on your
Maildir (for example, finding messages older than 3 months),
without parsing each file/message content.

From: Cyril RUSSO <boite.pour.spam@gmail.com>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-08-09 10:48:00 +04:00
Eygene Ryabinkin
41cb0f577f Prune trailing whitespaces from code and documentation
They are redundant in all pruned cases and sometimes even create some
problems, e.g., when one tries to jump through paragraphs in vi.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-07-21 23:00:23 +04:00
Dmitrijs Ledkovs
e8c40a9285 Merge branch 'master' into next 2013-07-10 02:49:52 +01:00
Adam Spiers
aa5202396c fix typos 2013-03-27 12:42:32 +00:00
Eygene Ryabinkin
f4140cbbed Create global instance of command-line options
This eases testing of option values inside the code.  This instance
is implemented as the read-only copy of the obtained 'options' object,
so callers won't be able to modify its contents.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-02-11 23:34:18 +04:00
X-Ryl669
5c842c01bd Fix deadlock for IMAP folder synced in singlethreaded mode
The problem lies in the fact that offlineimap.folder.Base's method
syncmessagesto_copy() uses threaded code everytime it is suggested by
the derived class's suggeststhreads() (currently, only IMAP does this
suggestion), but offlineimap/init.py will not spawn the
exitnotifymonitorloop() from offlineimap.threadutil.

The root cause is that ExitNotifyThread-derived threads need
offlineimap.threadutil's exitnotifymonitorloop() to be running the
cleaner for the exitthreads Queue(), because it fills the queue via
the run() method from this class: it wants to put() itself to the
Queue on exit, so when no exitnotifymonitorloop() is running, the
queue will fill up.  And if this thread is an instance of
InstanceLimitedThread that hits the limit on the number of threads,
then it will hold the instancelimitedsems[] semaphore will prevent
other InstanceLimitedThread()s of the same name to pass its start()
method.

The fix is to avoid using threaded code if we're running
single-threaded.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Obtained-from: X-Ryl669 <boite.pour.spam@gmail.com>
2013-02-05 07:53:25 +04:00
Dave Abrahams
e94642bb4d Determine folder syncing on Folder initialization
Rather than later when retrieving them by IMAP, we determine the
"sync_this" value of a folder on Folder() initialization.

This also fixes a bug where folder structure was not propagated when
a folder was filtered out but included in the folderincludes list.

Patch by Dave, split and modified slightly by Sebastian

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-09-01 02:30:46 +02:00
David Logie
a614f9a735 Fix str.format() calls for Python 2.6.
Python 2.6 requires a slightly different string formatting that >2.7.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-06-02 18:25:51 +02:00
Sebastian Spaeth
925538f02d docs: Fix docstrings to proper .rst syntax
Prevents compile errors when creating the user documentation.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-24 11:20:22 +01:00
Sebastian Spaeth
256a26a649 dry-run mode: Protect us from actually deleting a message in dry-run mode
Document which functions honor dry-run mode and which don't.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-17 13:17:05 +01:00
Sebastian Spaeth
5ef69e95c0 Prevent modifications on a folder level to occur in dry-run
Prevent savemessage(), and savemessageflags() to occur in dryrun mode in
all backends. Still need to protect against deletemessage().

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-17 13:17:05 +01:00
Sebastian Spaeth
10dd317026 Folder: Implement __eq__ for folders
This allows to compare folders directly with strings. It also allows
constructs such as "if 'moo' in repo.getfolders()".

See the code documentation for the exact behavior (it basically is equal if
it's the same instance *or* a string matching the untranslated folder name.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-16 11:03:33 +01:00
Sebastian Spaeth
81fc20c7ca Remove python<2.6 import workarounds (set & ssl)
'set' is builtin since python2.6, so remove the imports. Also 'ssl' exists
since 2.6 and has everything we need, so no need for conditional import
tests here anymore.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
0844d27f9f except Ex, e: --> except Ex as e:
Nudge us towards python3 compatability by converting deprecated python2 syntax.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
165cf5cb4f Use modern "with open() as file"
Saves a LOC and guarantees that the file will be closed

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:03:03 +01:00
Sebastian Spaeth
de1e03b065 Tweak code documentation
Minor comment changes

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:03:03 +01:00
Sebastian Spaeth
d3c693be7d Rename folder.saveuidvalidity()
Simply rename to folder.save_uidvalidity()

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:03:03 +01:00
Sebastian Spaeth
eb616da602 Rename folder.isuidvalidityok()
Use nicer name folder.check_uidvalidity() and improve code documentation

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:03:03 +01:00
Sebastian Spaeth
297a618441 Rename folder.getsaveduidvalidity(), folder.getuidvalidity()
Modernize names to: folder.get_saveduidvalidity(), folder.get_uidvalidity()

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:03:03 +01:00
Sebastian Spaeth
9453e1d955 Fix getuidvalidity crash (UIDVALIDITY returning None)
Rename getuidvalidity -> get_uidvalidity and cache the IMAP result.

1) Start modernizing our function names using more underscores
2) IMAPs implementation of get_uidvalidity was removing the UIDVALIDITY result
   from the imaplib2 result stack, so subsequent calls would return "None".
   As various functions can invoke this, this led to some errors that we
   avoid by caching the current UIDVALIDITY value in the Folder instance.

There are more simplifications and improvements to be made.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:02:13 +01:00
Sebastian Spaeth
0ea6c6ed47 Forgot to change a variable name in all cases
the newuid var was renamed new_uid but there was one leftover. Fixed.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 21:41:04 +01:00
Sebastian Spaeth
78a37f27ef WIP, revamp how we treat top-level dirs
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 19:08:48 +01:00
Sebastian Spaeth
de537dc09c Implement change_message_uid
Previously, assigning a new UID to a mapped IMAP or Maildir repository
was done by loading the "local" item, saving it under a new UID and
deleting the old one. This involved lots of disk activity for nothing
more than an effective file rename in Maildirs, and lots of network
usage in the MappedUID cases.

We do this on every upload from a local to a remote item, so that can
potentially be quite expensive. This patch lets backends that support it
(Maildir, MappedUID) efficiently rename the file rather than having to
read the mail content, write it out as a new file and delete the old
file. This speeds up uploads from Maildir and the MappedUID server.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 19:08:48 +01:00
Sebastian Spaeth
0ccf06d5e6 Implement clean CTRL-C termination
Previously, we would simply bail out in an ugly way, potentially leaving
temporary files around etc, or while writing status files. Hand SIGINT
and SIGTERM as an event to the Account class, and make that bail out
cleanly at predefined points. Stopping on ctrl-c can take a few seconds
(it will e.g. finish to transfer the ongoing message), but it will shut
down cleanly.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-04 19:31:27 +01:00
Sebastian Spaeth
f4a32bafd6 Pass ui.registerthread an Account() and not a name as string
This way, we can use all the account functions such as set_abort_event()
from the ui if needed.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-03 13:45:44 +01:00
Sebastian Spaeth
d54859a931 Don't setDaemon explicitly, it's done inherently
All ExitNotifyThreads and InstanceLimitThreads are setDaemon(True) in their
constructor, so there is no need to do that again in the code.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-02 11:29:23 +01:00
Sebastian Spaeth
eb0b546927 Output progress indication when copying files
Output (2 of 500) when logging message copying. This required moving of
self.ui.copyingmessage into a different function where we actually have
the information about the progress handy.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-29 16:51:48 +02:00
Sebastian Spaeth
e30ae53b2a Folder: Force top-level folder visiblename to ''
nametrans rules can lead to different visiblename names for the
top-level directory, specifically both '.' and '' (the latter was
recently introduced). However, we need to be able to compare folder
names to see if we need to create a new directory or whether a directory
already exists, so we need to be able to compare a repositories
visiblename (=transposed via nametrans rule) with another folder.

To make the top-level directory comparison happen, we enforce a
top-level name of '', so that comparisons work.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-23 14:31:49 +02:00
Sebastian Spaeth
6b2ec956cf Apply nametrans to all Foldertypes
getvisiblename() was only defined on IMAP(derived) foldertypes, but we
want it on eg. Maildirs too, so we define it centrally in Folder.Base.py
rather than only in folder.IMAP.py.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 14:23:08 +02:00
Sebastian Spaeth
8ba17c5bd1 add sync_this variable to all Folder() instances
This variable shows if this folder should be synced or is disabled due to
a folderfilter statement. This lets us distinguish between a non-existent
folder and one that has been filtered out. Previously any filtered folder
would simply appear to be non-existing.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 11:35:52 +02:00
Sebastian Spaeth
ee75e0921f Remove 'config' as parameter from BaseFolder & derivatives
It is possible to get the config parameter from the Repository() which is
set in BaseFolder, so we set self.config there and remove the various
methods and 'config' parameters that are superfluous.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-16 12:54:12 +02:00
Sebastian Spaeth
410e2d35e9 Set accountname in BaseFolder, and don't pass it in initialization
We passed in the accountname to all derivatives of BaseFolder, such as
IMAPFolder(...,repository,...,accountname), although it is perfectly
possible to get the accountname from the Repository(). So remove this
unneeded parameter. Each backend had to define getaccountname() (although
the function is hardly used and most accessed .accountname directly).

On the other hand BaseFolder was using getaccountname but it never defined
the function. So make the sane thing, remove all definitions from backends
and define accountname() once in Basefolder. It was made a property and not
just a (public) attribute, so it will show up in our developer
documentation as public API.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-16 12:54:12 +02:00
Sebastian Spaeth
c93cd9bb1a BaseFolder(): Save name and repository
As all Folders share these parameters, we can safely handle them in
BaseFolder. This makes sense, as BaseFolder has a getname() function
that returns self.name but nothing actually set self.name.

It also saves a few lines of code.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-16 12:54:12 +02:00
Sebastian Spaeth
a6480d4959 Fix string formatting
We were omitting an '%' where we needed it. Also include the traceback
information where it belongs in the new ui.error infrastructure.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-23 20:58:55 +02:00
Vladimir Marek
8de326b29f Catch correct type of exception
Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-19 19:05:16 +02:00
Sebastian Spaeth
466ded04d9 Make flags a set rather than a list
As this is essentially what it is, a set of values. This allows as
to do set arithmetics to see, e.g. the intersection of 2 flag sets
rather than clunkily having to do:

for flag in newflags:
  if flag not in oldflags:
    oldflags.append(flag)

Also some more code documenting.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-19 18:56:10 +02:00
Sebastian Spaeth
373e7cdbc1 Allow empty foldernames
Empty foldernames (as they could be created through nametrans) were
failing as the uidvalidity and status files names as determined by
folder/Base.py:getfolderbasename() lead to invalid file names ''.

Fix this by handling empty file names and translating them to '.' which
leads to the special file name 'dot'. (this special value existed before
and was not invented by this patch)

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-19 18:37:00 +02:00
Sebastian Spaeth
6f9b171ffd Don't pass a list to ui.copyingmessage()
We only copy to a single folder anyway, so clean up the code to only
pass in a single folder.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-17 20:41:32 +02:00
Sebastian Spaeth
9f23fea74e Fix error handling in folder.Base.copymessageto()
This is a bug fix on several levels. 1) We were lacking the import of
OfflineImapError. 2) OfflineImap.ERROR.MESSAGE was misspelled as ERROR.Message.
3) COntinuing with the next message only worked in single-thread mode
(using debug) and not in multi-thread mode. The reason is that we were
invoking a new thread and catching Exceptions in the main thread. But
python immediately aborts if an Exception bubbles up to a thread start.
This was fixed by catching exceptions directly in copymessageto() which
is the new thread, rather than catching them in the main thread.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 12:16:28 +02:00
Sebastian Spaeth
b47bc44783 accounts.py: Use ui.error when raising exceptions
Rather than using ui.warn, use ui.error() which outputs Exceptions to
the error log, saving them to a stack, so we get notified again at the
end of the sync run.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-11 19:14:28 +02:00
Vladimir Marek
d5cbdc4c0e Handle when UID can't be found on saved messages
Message was stored to dstfolder, but we can't find it's UID. This means we can't
link current message to the one created in IMAP. So we just delete local message
and on next run we'll sync it back. Also fixed imap.savemessage description.

This was broken by e20d8b9679.

Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-07-27 18:54:53 +02:00
Nicolas Sebrecht
136237b7dc refactoring: simplify the semaphorewait logic
The semaphorewait()/waitforthread() logic is usefull for IMAP starting
connections. We actually use it in imapserver only.

This patch removes the over-engineered factorized methods. It tend to simplify
the code by cleaning out a chain of two direct calls with no other processes.

Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-12 18:30:39 +02:00
Sebastian Spaeth
2fc12e875a Experimental LocalStatus stored in SQLite database
Based on patches by Stewart Smith, updated by Rob Browning.

plus:
- Inherit LocalStatusSQLFolder from LocalStatusFolder
  This lets us remove all functions that are available via our ancestors
  classes and are not needed.

- Don't fail if pysql import fails. Fail rather at runtime when needed.

- When creating the db file, create a metadata table which contains the
  format version info, so we can upgrade nicely to other formats.

- Create an upgrade_db() function which allows us to upgrade from any
  previous file format to the current one (including plain text)

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:29:11 +02:00