Commit Graph

158 Commits

Author SHA1 Message Date
Valentin Lab
f2ca4217c6 fix: broken retry loop would break connection management
The retry loop would release connection that would get
re-released upon ``finally`` clause. In consequence, an
exception would be cast.

Signed-off-by: Valentin Lab <valentin.lab@kalysto.org>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-11-26 17:11:59 +01:00
Igor Almeida
73a3767d11 IMAP folder: expose the message keywords
The keywords are in the flag string, so imaputil can just strip the
usual \Flags.

Signed-off-by: Igor Almeida <igor.contato@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-11-22 19:51:02 +01:00
Nicolas Sebrecht
3daddb9b33 folder/IMAP: minor indent fix
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-10-13 00:45:51 +02:00
Tommie Gannert
a6e7b6627b Add decodefoldernames option to decode IMAP folder names using UTF-7.
Signed-off-by: Tommie Gannert <tommie@gannert.se>
2015-08-29 16:43:33 +01:00
Michael Hoy
8dd146d8fe folder/IMAP: fix datetuple dst check
Signed-off-by: Michael Hoy <mjh@mjhoy.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-04-09 01:50:18 +02:00
Janna Martl
8096f6cd5b maxage: fix timezone issues, remove IMAP-IMAP support, add startdate option
1. When using maxage, local and remote messagelists are supposed to only
contain messages from at most maxage days ago. But local and remote used
different timezones to calculate what "maxage days ago" means, resulting
in removals on one side. Now, we ask the local folder for maxage days'
worth of mail, find the lowest UID, and then ask the remote folder for
all UID's starting with that lowest one.

2. maxage was fundamentally wrong in the IMAP-IMAP case: it assumed that
remote messages have UIDs in the same order as their local counterparts,
which could be false, e.g. when messages are copied in quick succession.
So, remove support for maxage in the IMAP-IMAP case.

3. Add startdate option for IMAP-IMAP syncs: use messages from the given
repository starting at startdate, and all messages from the other
repository. In the first sync, the other repository must be empty.

4. Allow maxage to be specified either as number of days to sync (as
previously) or as a fixed date.

Signed-off-by: Janna Martl <janna.martl109@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-04-07 11:54:35 +02:00
Janna Martl
428349e3f4 fix inaccurate UI messages when some messages are excluded from the cached lists
Some messages were excluded from the copy/delete list after the UI message said
they were copied/deleted.
Also fix Internaldate2epoch(), which was wrong.

Signed-off-by: Janna Martl <janna.martl109@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-03-21 14:10:57 +01:00
Nicolas Sebrecht
9e63fa3784 fix: folder/*: never set self.messagelist to None
Empty the list by setting an empty dict.
Introduce BaseFolder().ismessagelistempty().

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-02-13 17:07:18 +01:00
Nicolas Sebrecht
461554b7b1 more style consistency
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-02-10 17:25:00 +01:00
Eygene Ryabinkin
6fc9c36014 Fix regression introduced in 0f40ca4799
We have no variable "fullname", it must have been slipped in
unintentionally.

Blame commit:
	0f40ca4799 more style consistency

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-12 19:55:05 +01:00
Giovanni Mascellani
29e9b7ab39 Drop caches after having processed folders.
This enhances a lot memory consumption when you have many folders
to process.

Signed-off-by: Giovanni Mascellani <mascellani@poisson.phc.unipi.it>
2015-01-12 17:17:42 +01:00
Nicolas Sebrecht
a44718130d minor: add comments
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-12 17:17:42 +01:00
Eygene Ryabinkin
e7fabf9e57 Properly re-raise exception to save original tracebacks
We usually mutate some exceptions to OfflineImapError() and it is
a whole lot better if such exception will show up with the original
traceback, so all valid occurrences of such mutations were transformed
to the 3-tuple form of "raise".  Had also added coding guidelines
document where this re-raise strategy is documented.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2015-01-12 17:51:21 +03:00
Nicolas Sebrecht
0f40ca4799 more style consistency
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-08 23:36:19 +01:00
Eygene Ryabinkin
7df765cfdb Properly manipulate contents of messagelist for folder
Create initializer function that puts default values to all fields
of message list item.  Fix all code that directly assigns some hash
to the elements of messagelist: for direct assignments only initializer
is now permitted, all other modification are done in-place.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-08-03 16:47:26 +04:00
Eygene Ryabinkin
aa55b38e26 Avoid copying array every time, just slice it
Suggested-by: Josh Berry
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-07-10 10:25:34 +04:00
Eygene Ryabinkin
1f2e8af8aa Trade recursion by plain old cycle
We can do a simpler and more stack-friendly hack for IMAP
servers with limited line lenghts.

Reported-by: Josh Berry, https://github.com/josh-berry
GH: https://github.com/OfflineIMAP/offlineimap/pull/100
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-07-08 12:23:05 +04:00
Eygene Ryabinkin
ffd1b1d691 IMAP: provide message-id in error messages
This is handy for debug purposes when one tries to locate exact
message that was e.g. rejected by server.

Feature-request: http://comments.gmane.org/gmane.mail.imap.offlineimap.general/6491
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-07-01 07:44:18 +04:00
Eygene Ryabinkin
e8db1217d4 Fix improper header separator for X-OfflineIMAP header
For servers without UIDPLUS we are inserting additional header
just after transformation '\n' -> CRLF was done.  addmessageheaders()
was written to work with just '\n' as the separator, so X-OfflineIMAP
header wasn't preceeded by the CRLF, but just by '\n'.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-06-01 22:12:26 +04:00
Abdo Roig-Maranges
8c43e52173 Fix multiple typos in var, function and exception names
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-05-07 18:29:47 +04:00
Abdó Roig-Maranges
0e4afa9132 Make GmailFolder sync GMail labels
When synclabels config flag is set to "yes" for the GMail repo,
offlineimap fetches the message labels along with the messages, and
embeds them into the body under the header X-Keywords (or whatever
'labelsheader' was set to), as a comma separated list.

It also adds an extra pass to savemessageto, that performs label
synchronization on existing messages from GMail to local, the same way
it is done with flags.

We also introduce GmailMaildir repository that adds functionality to
change message labels.  It keeps track of messages modification time,
so one can quickly detect when the labels may have changed.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-05-06 23:36:06 +04:00
Abdó Roig-Maranges
9319ae212b Restructured folder/IMAP code
In preparation for GMail label sync, we had split our some functionality
that will be needed further into their own functions.  This also permitted
the code to look more compact and concise.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-05-06 23:03:21 +04:00
Abdó Roig-Maranges
5391476dfb Add ability to trim some local mail headers
When filterheaders is set to a comma-separated list of headers,
OfflineIMAP removes those headers from messages before uploading them
to the server.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-05-03 14:45:28 +04:00
Eygene Ryabinkin
844ca6b08c Don't embed CRLF multiple times when saving a message
Since we just do multiple passes for saving the message without
actually modifying its content (apart from header insertion that
is CRLF-clean), we can change line ends to the proper CRLF just
once.

And we can also get message's date only once too.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-05-03 14:43:11 +04:00
Eygene Ryabinkin
6cbd2498ae Refactoring: make functions to be private if we can
Make external API of class/module to be smaller, explicitely mark
all internal functions.  Also annotate methods that are implemented
as the part of the parent class interface.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-04-30 19:35:56 +04:00
Eygene Ryabinkin
b01274ce38 Fix double release of IMAP connection object
This commit fixes the case when we're invoking releaseconnection()
for a given imapobj twice.

This bug manifests itself as
{{{
ValueError: list.remove(x): x not in list

  File "[...]/offlineimap/folder/IMAP.py", line 615, in savemessage
    self.imapserver.releaseconnection(imapobj)
  File "[...]/offlineimap/imapserver.py", line 130, in releaseconnection
    self.assignedconnections.remove(connection)
}}}

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-02-26 17:51:48 +04:00
X-Ryl669
3bc66c0858 Add support for alternative message date synchronisation
Global or per-repository option utime_from_message tells OfflineIMAP
to set file modification time of messages pushed from one repository
to another basing on the message's "Date" header.

This is useful if you are doing some processing/finding on your
Maildir (for example, finding messages older than 3 months),
without parsing each file/message content.

From: Cyril RUSSO <boite.pour.spam@gmail.com>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-08-09 10:48:00 +04:00
Adam Spiers
d39a1f864f make savemessage() handle NO response to APPEND correctly
IMAP servers can return `NO` responses to the `APPEND` command,
e.g. here's an example response from Groupwise's IMAP server:

    NO APPEND The 1500 MB storage limit has been exceeded.

In this case, savemessage() should abort the repository sync rather
than returning UID 0 which would cause the local copy of the message
being saved to get irreversibly deleted.

Signed-off-by: Adam Spiers <offlineimap@adamspiers.org>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-08-02 16:41:54 +04:00
Eygene Ryabinkin
41cb0f577f Prune trailing whitespaces from code and documentation
They are redundant in all pruned cases and sometimes even create some
problems, e.g., when one tries to jump through paragraphs in vi.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-07-21 23:00:23 +04:00
Dmitrijs Ledkovs
e8c40a9285 Merge branch 'master' into next 2013-07-10 02:49:52 +01:00
Adam Spiers
aa5202396c fix typos 2013-03-27 12:42:32 +00:00
Eygene Ryabinkin
611f6e89c0 IMAP class: don't suggest multithreading in single-threaded mode
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-02-11 23:34:28 +04:00
Tobias Thierer
caef9a72fc Fix IMAP folder throwing away time zone when parsing email Date headers
Fix imapfolder.getmessageinternaldate misparsing the Date:
header from emails due to a bug or surprising behaviour by
email.utils.parsedate. This is because email.utils.parsedate's
return value contains the unadjusted hour value from the string
parsed but does not include information about the time zone in
which it is specified. For example (Python 2.7.3):

$ python -c "import email.utils;
  print email.utils.parsedate('Mon, 20 Nov 1995 19:12:08 -0500')"
 (1995, 11, 20, 19, 12, 8, 0, 1, -1)

(the -1 is the isdst field); the -0500 time zone is completely
ignored, so e.g. the same input with time "19:12:08 +0300" has
the same result. When passed to time.struct_time as allowed per
the parsedate documentation, this time is interpreted in GMT and
thus deviates from the correct value by the timezone offset
(in this example, -5 hours).

I consider this a bug in email.utils.parsedate: In my opinion,
since the return value of the parsetime doesn't include a timezone,
it should be expressed in terms of UTC rather than in terms of the
time zone from the Date header; the existence of
email.utils.parsedate_tz, to which I've switched, indicates that
maybe the authors were aware of this problem.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-10-03 11:16:13 +02:00
Sebastian Spaeth
36156fa985 Remove the APPENDUID hack, previously introduced
As Gmail was only announcing the presence of the UIDPLUS extension
after we logged in, and we were then only getting server
capabilities before, a hack was introduced that checked the
existence of an APPENDUID reply, even if the server did not claim
to support it.

However, John Wiegley reports problems, where the APPENDUID would
be None, and we attempt to go this path (it seems that imaplib2
returns [None] if there is no such reply, so our test here for "!="
might fail. Given that this is an undocumented imaplib2 function
anyway, and we do fetch gmail capabilities after authentication,
this hack should no longer be necessary.

We had problems there earlier, where imapobj.response() would
return [None] although we had received a APPENDUID response from
the server, this might need more debugging and testing.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-06-06 10:13:42 +02:00
Sebastian Spaeth
61e754c65e Do not try to release IMAP connection twice
Reported by sharat87 in https://github.com/spaetz/offlineimap/pull/38,
he would often get an unhandled Exception when trying to
releaseconnection() a connection that was not in the pool of
connections.

The reason this could happen is that when folder.IMAP.quickchanged()
raises an Exception in select(), we would release the connection in the
"except" handling, and than release the same connection in the "finally"
clause, which led to the error. The right thing is to only release the
connection once, of course.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-04-21 13:32:14 +02:00
Sebastian Spaeth
0e6b4ae798 Revert "Clean up and improve APPENDUID handling"
This reverts commit 4d47f7bf3c.

This is one of two candidates for introducing the instabilities that
John Wiegley observed. We need to reintroduce with careful testing only.

The original patch has been mostly reverted.
2012-02-24 08:35:59 +01:00
Sebastian Spaeth
bc73c11239 Revert "Don't CHECK imapserver after each APPEND"
This reverts commit 47390e03d6.

It is one of two potential candidates for the APPENDUID
regression that John Wiegley reported. We need to examine this
carefully before reintroducing this patch.

Resolved Changelog.draft.rst conflict.
2012-02-24 08:31:31 +01:00
Sebastian Spaeth
5ef69e95c0 Prevent modifications on a folder level to occur in dry-run
Prevent savemessage(), and savemessageflags() to occur in dryrun mode in
all backends. Still need to protect against deletemessage().

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-17 13:17:05 +01:00
Sebastian Spaeth
7da50e638d folder/IMAP: better error when savemessage fails
If we cannot identify the new UID after a sendmessage(), log a better error
message, including the server response for better debugging.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-16 16:49:06 +01:00
Sebastian Spaeth
5c598d7e74 dict.has_key(a) --> a in dict
has_key() is gone in python3, so use a more modern syntax here.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:43 +01:00
Sebastian Spaeth
e6e708ec78 Don't use sort() on dict values()
This won't work in python3 anymore, so just use sorted() when needed.
In one case, we could remove the sort() completely as were were sanity checking
one line above, that we only having one UID as response which makes sorting
unneeded.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:43 +01:00
Sebastian Spaeth
b33f2452f0 Use "from . import" for relative imports
Will fail in python3 otherwise.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:43 +01:00
Sebastian Spaeth
81fc20c7ca Remove python<2.6 import workarounds (set & ssl)
'set' is builtin since python2.6, so remove the imports. Also 'ssl' exists
since 2.6 and has everything we need, so no need for conditional import
tests here anymore.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
c5468ae599 raise Exception, "text" --> raise Exception("text")
To have the code work in python3, we need to convert all occurences of

raise Exception, "text" to be proper functions. This style also adheres to PEP8.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
0844d27f9f except Ex, e: --> except Ex as e:
Nudge us towards python3 compatability by converting deprecated python2 syntax.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
47390e03d6 Don't CHECK imapserver after each APPEND
Most servers support the UIDPLUS extension, and we don't have to search
headers after each uploaded message. There is no need to CHECK the imap
server after each message when there is no need to search headers.

I have not measured the performance impact on real world servers, but
this lets us do less unneeded work in the common case.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-20 14:43:12 +01:00
Sebastian Spaeth
4d47f7bf3c Clean up and improve APPENDUID handling
We were not cleaning out possibly existing APPENDUID messages before
APPENDing a new message. In case an old message were still hanging
around, this could *possibly* lead to retrieving and old UID. Things
should have been fine, but we do want to play safe here.

Also, make use of the "official" imaplib2 .response() command rather
than the internal _get_untagged_response() function.

Remove the hack that we would be looking for APPENDUID responses even if
the server claimed not to support the UIDPLUS ext. We now poll server
CAPABILITIES after login, and Gmail does provide us with the UIDPLUS
capability after login.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-20 14:33:16 +01:00
Sebastian Spaeth
9453e1d955 Fix getuidvalidity crash (UIDVALIDITY returning None)
Rename getuidvalidity -> get_uidvalidity and cache the IMAP result.

1) Start modernizing our function names using more underscores
2) IMAPs implementation of get_uidvalidity was removing the UIDVALIDITY result
   from the imaplib2 result stack, so subsequent calls would return "None".
   As various functions can invoke this, this led to some errors that we
   avoid by caching the current UIDVALIDITY value in the Folder instance.

There are more simplifications and improvements to be made.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:02:13 +01:00
Sebastian Spaeth
3284e010ff Revert "use .response() rather _get_untagged_response()"
Recently the internal function use of imaplib2's _get_untagged_response()
was switched to use the public documented .response() function (which
should return the same data). However within a few fays we received reports
that both uses of a) the UIDVALIDITY fetching and b) the APPENDUID fetching
returned [None] as data although the IMAP log definitely shows that data
was returned. Revert to using the undocumented internal imaplib2 function,
that seemed to have worked without problems. This needs to be taken up to
the imaplib2 developer.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-09 09:51:43 +01:00
Sebastian Spaeth
7184ec28cc Sanity check return value of UIDVALIDTY response
We have a reported case where response('UIDVALIDITY') returned [None]
which results in an ugly non-intuitive crash. Sanity check and report
something nicer.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-08 12:26:47 +01:00