This should significantly improve performance when used to write large
amounts of messages.
This feature is enabled through the fsync configuration option.
Code refactorize around fsync.
This addresses #390 (although it doesn't necessarily fix all instances
of that problem yet).
Github-ref: https://github.com/OfflineIMAP/offlineimap/issues/390
Originally-written-by: Giel van Schijndel <me@mortis.eu>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
There is no other way to make Python2 and Python3 happy, because syntax
raise E, V, T is incompatible with the latter.
Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Python3 accepts binary input for md5 function.
This patch is known to break setups using folder names not strictly conforming
the IMAP UTF-7 encoding. We always made it clear that such setup is unsupported
and might be broken at some point in time. See documentation about
'decodefoldernames' in the provided configuration file. This is why this patch
is considered introducing no regression for this use case.
Patches to support both Python 3 and Python 2 by re-encoding the MD5 in the
filenames are welcome. This likely requires a new CLI option to allow
backporting the feature for users downgrading or changing of Python environment.
Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
There is no such method in Python3 any more.
Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
It is more readable and returns a list therefore it is compatible both
with Python 2 and 3.
Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Introduce the '--migrate-fmd5-using-nametrans' option which migrates the
FMD5 hashes from versions prior to 6.3.5.
It seems that commit 'Apply nametrans to all Foldertypes' (6b2ec956cf)
introduced a regression because it changed the FMD5 part of the filename
calculated by OfflineIMAP. Thus, OfflineIMAP believes that the messages
has been removed and adds them back.
For more information, see:
http://www.offlineimap.org/configuration/2016/02/12/debian-upgrade-from-jessie-to-stretch.html
Bug-Debian: https://bugs.debian.org/812108
Reported-by: François <francois@avalenn.eu>
Signed-off-by: Ilias Tsitsimpis <i.tsitsimpis@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Since this is used in an except calse, we first don't mask the real cause and
raise the original error.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Remove filtering that was previously done to avoid errors in flag
handling.
Signed-off-by: Igor Almeida <igor.contato@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
If this value is true, use (if possible) a timestamp based on message
Date or Delivery-date headers. The current system time is used
otherwise.
filename_use_mail_timestamp and utime_from_header are now completely
separated option that do not interfere one with other.
To handle this feature in a multithread context we use a hash to count
the number of mail with the same timestamp. This method is more accurate
than using the old lasttime and timeseq variables.
Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Handle case where email's internal time is erroneously so large as to
cause overflow errors when setting file modification time with
utime_from_header = true.
Reported-by: Cameron Simpson <cs@zip.com.au>
Signed-off-by: Janna Martl <janna.martl109@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
1. When using maxage, local and remote messagelists are supposed to only
contain messages from at most maxage days ago. But local and remote used
different timezones to calculate what "maxage days ago" means, resulting
in removals on one side. Now, we ask the local folder for maxage days'
worth of mail, find the lowest UID, and then ask the remote folder for
all UID's starting with that lowest one.
2. maxage was fundamentally wrong in the IMAP-IMAP case: it assumed that
remote messages have UIDs in the same order as their local counterparts,
which could be false, e.g. when messages are copied in quick succession.
So, remove support for maxage in the IMAP-IMAP case.
3. Add startdate option for IMAP-IMAP syncs: use messages from the given
repository starting at startdate, and all messages from the other
repository. In the first sync, the other repository must be empty.
4. Allow maxage to be specified either as number of days to sync (as
previously) or as a fixed date.
Signed-off-by: Janna Martl <janna.martl109@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We were using rtime for two different purposes:
- to store remote internal date
- to use in the utime_from_header option
Let's decouple the utime_from_header logic from rtime, now rtime means
remote internal date.
Signed-off-by: Abdo Roig-Maranges <abdo.roig@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This partially reverts commit 25513e9038.
Only changes about dates and times are reverted. The changes about the style are
kept.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Some messages were excluded from the copy/delete list after the UI message said
they were copied/deleted.
Also fix Internaldate2epoch(), which was wrong.
Signed-off-by: Janna Martl <janna.martl109@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Suppose messages A and B were delivered to the remote folder at
"maxage + 1" days ago.
A was downloaded to the local folder "maxage + 1" days ago, but B was only
downloaded "maxage - 1" days ago (contrived scenario to illustrate the two
things that could happen). The behavior was that B gets deleted from the local
folder, but A did not. The expected behavior is that neither is deleted.
Starting where Base.py: __syncmessagesto_delete(self, dstfolder, statusfolder)
is called where:
- self is the remote folder
and
- dstfolder is the local folder.
It defines deletelist to be the list of messages in the status folder
messagelist that aren't in the remote folder messagelist with
not self.uidexists(uid)
A and B are both in the status folder. They're also both *NOT* in the remote
folder messagelist: this list is formed in IMAP.py: cachemessagelist(), which
calls _msgs_to_fetch(), which only asks the IMAP server for messages that are
"< maxage" days old.
Back to Base.py __syncmessagesto_delete(), look at the call
folder.deletemessages(deletelist), where folder is the local folder. This ends
up calling Maildir.py deletemessage() for each message on the deletelist. But we
see that this methods returns (instead of deleting anything) if the message is
in the local folder's messagelist. This messagelist was created by Maildir.py's
cachemessagelist(), which calls _scanfolder(), which tries to exclude messages
based on maxage. So at this point, we *WANT* A and B to be excluded -- then they
will be spared from deletion. This maxage check calls _iswithinmaxage(), and
actually does the date comparison based on the time found at the beginning of
the message's filename. These filenames were originally created in Maildir.py's
new_message_filename(), which calls _gettimeseq() to get the current time (i.e.
the time of retrieval).
Upshot: A's filename has an older timestamp than B's filename. A is excluded
from the local folder messagelist in _scanfolder(), hence spared from deletion
in deletemessage(); B is not excluded, and is deleted.
This patch does not address the timezone issue. As for the IMAP/timezone issue,
a similar issue is discussed in the thunderbird bug tracker here:
https://bugzilla.mozilla.org/show_bug.cgi?id=886534
In the end, they're solving a different problem, but they agree that
there is really no reliable way of guessing the IMAP server's internal
timezone.
Signed-off-by: Janna Martl <janna.martl109@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We usually mutate some exceptions to OfflineImapError() and it is
a whole lot better if such exception will show up with the original
traceback, so all valid occurrences of such mutations were transformed
to the 3-tuple form of "raise". Had also added coding guidelines
document where this re-raise strategy is documented.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Create initializer function that puts default values to all fields
of message list item. Fix all code that directly assigns some hash
to the elements of messagelist: for direct assignments only initializer
is now permitted, all other modification are done in-place.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
When synclabels config flag is set to "yes" for the GMail repo,
offlineimap fetches the message labels along with the messages, and
embeds them into the body under the header X-Keywords (or whatever
'labelsheader' was set to), as a comma separated list.
It also adds an extra pass to savemessageto, that performs label
synchronization on existing messages from GMail to local, the same way
it is done with flags.
We also introduce GmailMaildir repository that adds functionality to
change message labels. It keeps track of messages modification time,
so one can quickly detect when the labels may have changed.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Make external API of class/module to be smaller, explicitely mark
all internal functions. Also annotate methods that are implemented
as the part of the parent class interface.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
They are redundant in all pruned cases and sometimes even create some
problems, e.g., when one tries to jump through paragraphs in vi.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Prevent savemessage(), and savemessageflags() to occur in dryrun mode in
all backends. Still need to protect against deletemessage().
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
There is no need to cast 0 to 'long' even if we want to compare it to long
numbers in modern pythons.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rename getuidvalidity -> get_uidvalidity and cache the IMAP result.
1) Start modernizing our function names using more underscores
2) IMAPs implementation of get_uidvalidity was removing the UIDVALIDITY result
from the imaplib2 result stack, so subsequent calls would return "None".
As various functions can invoke this, this led to some errors that we
avoid by caching the current UIDVALIDITY value in the Folder instance.
There are more simplifications and improvements to be made.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Do not read in custom maildir flags, or we would try to sync them over
the wire. The next step will be to merge flag writes with existing
custom flags, so we don't lose information.
The long term goal will be to attempt to sync flags to the other side,
of course.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rather than always parsing the filename, we only need to do so if the flags
have actually changed, otherwise we can keep the filename.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
