Commit Graph

335 Commits

Author SHA1 Message Date
Nicolas Sebrecht
3f70d9ef37 sqlite: open database when we use it rather than at instantiation time
Backported from 8e995a69bfa003ab822b55731429d84b3bc5626f.

We currently close the database as soon as possible while we handle the status
backend but this is still too early because autorefresh induces looping on this
code block.

Instead of delaying the closing outside of the loop, it's easier to delay the
opening as late as possible (inside the loop). The downside is that the database
is opened/closed more than once when autorefresh is enabled. The good news is
that this make the code much easier.

Fixes regression introduces by 6fb5700.

Reported-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-06-08 01:59:39 +02:00
Nicolas Sebrecht
b840e66d59 SQLite: close db when done
Backported from 6fb5700f94.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-06-08 01:59:14 +02:00
Ilias Tsitsimpis
c84d23b656 Identify and fix messages with FMD5 inconsistencies
Introduce the '--migrate-fmd5-using-nametrans' option which migrates the
FMD5 hashes from versions prior to 6.3.5.

It seems that commit 'Apply nametrans to all Foldertypes' (6b2ec956cf)
introduced a regression because it changed the FMD5 part of the filename
calculated by OfflineIMAP. Thus, OfflineIMAP believes that the messages
has been removed and adds them back.

For more information, see:
http://www.offlineimap.org/configuration/2016/02/12/debian-upgrade-from-jessie-to-stretch.html

Bug-Debian: https://bugs.debian.org/812108
Reported-by: François <francois@avalenn.eu>
Signed-off-by: Ilias Tsitsimpis <i.tsitsimpis@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-03-06 21:01:51 +01:00
Nicolas Sebrecht
36375daee6 fix: exceptions.OSError might not have attribute EEXIST defined
Since this is used in an except calse, we first don't mask the real cause and
raise the original error.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-02-19 12:33:57 +01:00
Nicolas Sebrecht
452ada330a folder/Maildir: ignore dot files
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-12-05 12:49:41 +01:00
Valentin Lab
f2ca4217c6 fix: broken retry loop would break connection management
The retry loop would release connection that would get
re-released upon ``finally`` clause. In consequence, an
exception would be cast.

Signed-off-by: Valentin Lab <valentin.lab@kalysto.org>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-11-26 17:11:59 +01:00
Nicolas Sebrecht
d6077a09cf Keywords: avoid warning at each message when no keywords are used
This fix does not apply when any keyword in configured which is already
harmless.

Written-by: Igor Almeida <igor.contato@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-11-22 20:05:34 +01:00
Igor Almeida
58a6f8b401 __syncmessagesto_flags: refactor for readability
Extract the flag/keyword translation and combination logic to a
function.

Signed-off-by: Igor Almeida <igor.contato@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-11-22 19:51:33 +01:00
Igor Almeida
61ee6e783e __syncmessagesto_flags: store keywords
This uses the destination folder's keyword mapping to translate the
message's keywords into some appropriate format.

Tested only with local Maildir.

Signed-off-by: Igor Almeida <igor.contato@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-11-22 19:51:20 +01:00
Igor Almeida
73a3767d11 IMAP folder: expose the message keywords
The keywords are in the flag string, so imaputil can just strip the
usual \Flags.

Signed-off-by: Igor Almeida <igor.contato@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-11-22 19:51:02 +01:00
Igor Almeida
c52ca66874 Maildir folder: extract lower-case letters (custom flags) from filename
Remove filtering that was previously done to avoid errors in flag
handling.

Signed-off-by: Igor Almeida <igor.contato@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-11-22 19:50:22 +01:00
Sébastien Gross
3b30c4aa93 add new config option filename_use_mail_timestamp
If this value is true, use (if possible) a timestamp based on message
Date or Delivery-date headers. The current system time is used
otherwise.

filename_use_mail_timestamp and utime_from_header are now completely
separated option that do not interfere one with other.

To handle this feature in a multithread context we use a hash to count
the number of mail with the same timestamp. This method is more accurate
than using the old lasttime and timeseq variables.

Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-11-04 00:56:46 +01:00
Nicolas Sebrecht
3daddb9b33 folder/IMAP: minor indent fix
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-10-13 00:45:51 +02:00
Janna Martl
ee0de28cc4 utime_from_header: handle out-of-bounds dates
Handle case where email's internal time is erroneously so large as to
cause overflow errors when setting file modification time with
utime_from_header = true.

Reported-by: Cameron Simpson <cs@zip.com.au>
Signed-off-by: Janna Martl <janna.martl109@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-10-12 16:19:23 +02:00
Nicolas Sebrecht
55ca922dd1 Merge branch 'krafczyk-new-mail-hook-2' into next 2015-09-11 21:04:52 +02:00
Matthew Krafczyk
48bb2f4113 Added the newmail_hook
When new mail arrives, this hook is triggered, allowing the user to
play a sound, or launch a popup.

Signed-off-by: Matthew Krafczyk <krafczyk.matthew@gmail.com>
2015-09-11 20:34:38 +02:00
Tommie Gannert
a6e7b6627b Add decodefoldernames option to decode IMAP folder names using UTF-7.
Signed-off-by: Tommie Gannert <tommie@gannert.se>
2015-08-29 16:43:33 +01:00
Michael Hoy
8dd146d8fe folder/IMAP: fix datetuple dst check
Signed-off-by: Michael Hoy <mjh@mjhoy.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-04-09 01:50:18 +02:00
Nicolas Sebrecht
a9ed2ff98a folder/Maildir: fix emailutil import
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-04-07 11:57:30 +02:00
Janna Martl
8096f6cd5b maxage: fix timezone issues, remove IMAP-IMAP support, add startdate option
1. When using maxage, local and remote messagelists are supposed to only
contain messages from at most maxage days ago. But local and remote used
different timezones to calculate what "maxage days ago" means, resulting
in removals on one side. Now, we ask the local folder for maxage days'
worth of mail, find the lowest UID, and then ask the remote folder for
all UID's starting with that lowest one.

2. maxage was fundamentally wrong in the IMAP-IMAP case: it assumed that
remote messages have UIDs in the same order as their local counterparts,
which could be false, e.g. when messages are copied in quick succession.
So, remove support for maxage in the IMAP-IMAP case.

3. Add startdate option for IMAP-IMAP syncs: use messages from the given
repository starting at startdate, and all messages from the other
repository. In the first sync, the other repository must be empty.

4. Allow maxage to be specified either as number of days to sync (as
previously) or as a fixed date.

Signed-off-by: Janna Martl <janna.martl109@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-04-07 11:54:35 +02:00
Nicolas Sebrecht
71693b7d8c Maildir: fix imports of emailutil
Commit 0c17350e4f revisited utime_for_header without taking care of the
imports.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-04-06 23:50:56 +02:00
Abdo Roig-Maranges
c094304f93 make savemessagelabels honor utime_from_header
Previously, syncing labels on a message always resulted in updating the
file modification time, even with utime_from_headers=true

This patch restores the file mtime to the previous value when
utime_from_headers=true, preventing a label synchronization from
breaking the promise that the file mtimes coincide with the header date.

Signed-off-by: Abdo Roig-Maranges <abdo.roig@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-04-06 17:49:36 +02:00
Abdo Roig-Maranges
0c17350e4f decouple utime_from_header from rtime
We were using rtime for two different purposes:
- to store remote internal date
- to use in the utime_from_header option

Let's decouple the utime_from_header logic from rtime, now rtime means
remote internal date.

Signed-off-by: Abdo Roig-Maranges <abdo.roig@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-04-03 11:30:30 +02:00
Nicolas Sebrecht
5a4e4fd0c2 rename utime_from_message to utime_from_header
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-03-27 16:50:17 +01:00
Nicolas Sebrecht
05fa311878 Revert "fix: don't loose local mails because of maxage"
This partially reverts commit 25513e9038.
Only changes about dates and times are reverted. The changes about the style are
kept.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-03-26 12:05:11 +01:00
Janna Martl
731129396c fix: don't copy messages if not necessary
Fix regresssion introduced in 428349e3.

Prevent messages with UID's already in the destination folder from getting
excluded from the copy list.

Signed-off-by: Janna Martl <janna.martl109@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-03-22 10:42:29 +01:00
Janna Martl
428349e3f4 fix inaccurate UI messages when some messages are excluded from the cached lists
Some messages were excluded from the copy/delete list after the UI message said
they were copied/deleted.
Also fix Internaldate2epoch(), which was wrong.

Signed-off-by: Janna Martl <janna.martl109@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-03-21 14:10:57 +01:00
Janna Martl
25513e9038 fix: don't loose local mails because of maxage
Suppose messages A and B were delivered to the remote folder at
"maxage + 1" days ago.

A was downloaded to the local folder "maxage + 1" days ago, but B was only
downloaded "maxage - 1" days ago (contrived scenario to illustrate the two
things that could happen). The behavior was that B gets deleted from the local
folder, but A did not. The expected behavior is that neither is deleted.

Starting where Base.py: __syncmessagesto_delete(self, dstfolder, statusfolder)
is called where:
 - self is the remote folder
and
 - dstfolder is the local folder.

It defines deletelist to be the list of messages in the status folder
messagelist that aren't in the remote folder messagelist with

  not self.uidexists(uid)

A and B are both in the status folder. They're also both *NOT* in the remote
folder messagelist: this list is formed in IMAP.py: cachemessagelist(), which
calls _msgs_to_fetch(), which only asks the IMAP server for messages that are
"< maxage" days old.

Back to Base.py __syncmessagesto_delete(), look at the call
folder.deletemessages(deletelist), where folder is the local folder. This ends
up calling Maildir.py deletemessage() for each message on the deletelist. But we
see that this methods returns (instead of deleting anything) if the message is
in the local folder's messagelist. This messagelist was created by Maildir.py's
cachemessagelist(), which calls _scanfolder(), which tries to exclude messages
based on maxage. So at this point, we *WANT* A and B to be excluded -- then they
will be spared from deletion. This maxage check calls _iswithinmaxage(), and
actually does the date comparison based on the time found at the beginning of
the message's filename. These filenames were originally created in Maildir.py's
new_message_filename(), which calls _gettimeseq() to get the current time (i.e.
the time of retrieval).

Upshot: A's filename has an older timestamp than B's filename. A is excluded
from the local folder messagelist in _scanfolder(), hence spared from deletion
in deletemessage(); B is not excluded, and is deleted.

This patch does not address the timezone issue. As for the IMAP/timezone issue,
a similar issue is discussed in the thunderbird bug tracker here:

https://bugzilla.mozilla.org/show_bug.cgi?id=886534

In the end, they're solving a different problem, but they agree that
there is really no reliable way of guessing the IMAP server's internal
timezone.

Signed-off-by: Janna Martl <janna.martl109@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-03-09 10:31:54 +01:00
Nicolas Sebrecht
efc4df1bd7 LocalStatusSQLite: labels: don't fail if database returns unexpected None value
This requires more researches. See
  https://github.com/OfflineIMAP/offlineimap/issues/103
.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-02-22 14:15:55 +01:00
Nicolas Sebrecht
a1383da9b3 LocalStatusSQLite: provide information on what is failing for OperationalError
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-02-16 10:31:21 +01:00
Nicolas Sebrecht
73952b8c2c offlineimap.conf: say what is the default value for the sep option
Some style improvements.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-02-13 17:43:29 +01:00
Nicolas Sebrecht
655c2b1fb9 sqlite: provide offending filename when open fails
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-02-13 17:18:07 +01:00
Nicolas Sebrecht
9e63fa3784 fix: folder/*: never set self.messagelist to None
Empty the list by setting an empty dict.
Introduce BaseFolder().ismessagelistempty().

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-02-13 17:07:18 +01:00
Nicolas Sebrecht
baee2b6fd9 fix: folder/Base: missing """ for doc string
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-02-13 17:07:18 +01:00
Nicolas Sebrecht
95eb8697f9 folder: LocalStatus: avoid to redefine 'file'
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-02-10 17:35:44 +01:00
Nicolas Sebrecht
461554b7b1 more style consistency
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-02-10 17:25:00 +01:00
Nicolas Sebrecht
cb6790b804 minor: fix indentation
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-18 21:52:09 +01:00
Nicolas Sebrecht
d0cb96781c folder: LocalStatus(SQLite): avoid redefining unchanged Base method
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-18 21:52:09 +01:00
Nicolas Sebrecht
2af3d93a85 folder: LocalStatusSQLite: remove unused import
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-18 21:52:09 +01:00
Nicolas Sebrecht
285295c4f2 folder: LocalStatus: revamp cachemessagelist()
- Do not redefine "file".
- break loop as soon as possible.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-18 21:52:09 +01:00
Eygene Ryabinkin
6fc9c36014 Fix regression introduced in 0f40ca4799
We have no variable "fullname", it must have been slipped in
unintentionally.

Blame commit:
	0f40ca4799 more style consistency

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-12 19:55:05 +01:00
Wieland Hoffmann
d08f6d15c2 addmessageheader: Add a note about the incorrect rendering of the docstring
The note tells people to look at the source of the method, which
spinx.ext.viewcode conveniently links right next to the methods
signature.

Signed-off-by: Wieland Hoffmann <themineo@gmail.com>
2015-01-12 17:17:42 +01:00
Giovanni Mascellani
29e9b7ab39 Drop caches after having processed folders.
This enhances a lot memory consumption when you have many folders
to process.

Signed-off-by: Giovanni Mascellani <mascellani@poisson.phc.unipi.it>
2015-01-12 17:17:42 +01:00
Nicolas Sebrecht
a44718130d minor: add comments
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-12 17:17:42 +01:00
Eygene Ryabinkin
e7fabf9e57 Properly re-raise exception to save original tracebacks
We usually mutate some exceptions to OfflineImapError() and it is
a whole lot better if such exception will show up with the original
traceback, so all valid occurrences of such mutations were transformed
to the 3-tuple form of "raise".  Had also added coding guidelines
document where this re-raise strategy is documented.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2015-01-12 17:51:21 +03:00
Wieland Hoffmann
2f9d8d22ee folder/Base.py: fix comment: para -> param
Signed-off-by: Wieland Hoffmann <themineo@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-09 01:10:49 +01:00
Nicolas Sebrecht
0f40ca4799 more style consistency
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-08 23:36:19 +01:00
Nicolas Sebrecht
61021260cb more consistent style
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-07 21:31:43 +01:00
Abdo Roig-Maranges
e70607e3e3 Warn about a tricky piece of code in addmessageheader
As requested by Nicholas Sebrecht.

Signed-off-by: Abdo Roig-Maranges <abdo.roig@gmail.com>
2014-11-20 14:18:35 +01:00
Abdo Roig-Maranges
2a5ef8c2ef Delete gmail labels header before adding a new one
This fixes a bug in which a message ended up with multiple gmail labels
header (X-Keywords or so). Fix fix it by removing all labels headers
before adding the updated one.

Signed-off-by: Abdo Roig-Maranges <abdo.roig@gmail.com>
2014-11-20 14:16:48 +01:00