WARNING: I consider the Gmail "realdelete" option as harmful with the
potential for DATALOSS. Add scary warnings to offlineimap.conf.
See the analysis at
http://article.gmane.org/gmane.mail.imap.offlineimap.general/5265
Deleting a message from a Gmail folder via the IMAP interface will just
remove that folder's label from the message: the message will continue
to exist in the '[Gmail]/All Mail' folder. If `realdelete` is set to
`True`, then deleted messages will be moved to the '[Gmail]/Trash'
folder. BEWARE: this will immediately delete a messages from *all
folders* it belongs to!
AS OFFLINEIMAP IMPLEMENTS FOLDER MOVES AS 1) AN ADD and 2) A DELETE (the
order can vary), THIS MEANS THAT A FOLDER MOVE CAN CAUSE DATALOSS. DO
NOT USE IT AND MOVE MAIL TO "[Gmail]/Trash" TO DELETE MAIL FROM
"[Gmail]/All Mail"!
We will need to discuss whether to completely disable that feature.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
WARNING: I consider the Gmail "realdelete" option as harmful with the
potential for DATALOSS. Add scary warnings to offlineimap.conf.
See the analysis at
http://article.gmane.org/gmane.mail.imap.offlineimap.general/5265
Deleting a message from a Gmail folder via the IMAP interface will just
remove that folder's label from the message: the message will continue
to exist in the '[Gmail]/All Mail' folder. If `realdelete` is set to
`True`, then deleted messages will be moved to the '[Gmail]/Trash'
folder. BEWARE: this will immediately delete a messages from *all
folders* it belongs to!
AS OFFLINEIMAP IMPLEMENTS FOLDER MOVES AS 1) AN ADD and 2) A DELETE (the
order can vary), THIS MEANS THAT A FOLDER MOVE CAN CAUSE DATALOSS. DO
NOT USE IT AND MOVE MAIL TO "[Gmail]/Trash" TO DELETE MAIL FROM
"[Gmail]/All Mail"!
We will need to discuss whether to completely disable that feature.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Add a changelog to W. Trevor King's previous commit. Also make wording a
bit more consistent and and remove a now unneeded comparison (dirname is
always set when extension is set).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
On Tue, Dec 13, 2011 at 12:00:57PM -0500, W. Trevor King wrote:
> I've attached a patch that does fix the problem…
Oops, *now* I've attached the patch and logs ;).
--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
From 3067b1b4dfb00d165bd9480ea49f446adb12991d Mon Sep 17 00:00:00 2001
From: W. Trevor King <wking@drexel.edu>
Date: Tue, 13 Dec 2011 11:26:00 -0500
Subject: [PATCH] Only scan children in _getfolders_scandir if extension is set.
When sep is '/', MaildirRepository._getfolders_scandir recursively
checks sub-directories for additional maildirs. The old loop logic
always checked the top directory and its children. This lead to
children being found twice, once from their parent, with dirname
matching their directory name, and once from themselves, with a
dirname of ''.
This patch fixes the problem by only checking the top directory when
extension is not set (i.e. for the root directory).
Do not read in custom maildir flags, or we would try to sync them over
the wire. The next step will be to merge flag writes with existing
custom flags, so we don't lose information.
The long term goal will be to attempt to sync flags to the other side,
of course.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Blinkenlights UI 6.5.0 regression fixes only.
* Sleep led to crash ('abort_signal' not existing)
* Make exit via 'q' key work again cleanly
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
With the new abort signal handler, we can send a signal that lets us
exit cleanly. Make use of this, rather than crashing out in ugly ways.
This affects only the Blinkenlights UI when pressing 'q'.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This is a regression introduced when renaming signals due to the
improved CTRL-C handling. Regression in 6.5.0
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
OfflineIMAP v6.5.1 (2012-01-07) - "Quest for stability"
=======================================================
* Fixed Maildir regression "flagmatchre" not found. (regressed in 6.5.0)
* Have console output go by default to STDOUT and not STDERR (regression
in 6.5.0)
* Fixed MachineUI to urlencode() output lines again, rather than
outputting multi-line items. It's ugly as hell, but it had been that
way for years.
* Remove the old global locking system. We lock only the accounts that
we currently sync, so you can invoke OfflineImap multiple times now as
long as you sync different accounts. This system is compatible with
all releases >= 6.4.0, so don't run older releases simultanous to this
one.
THe new logging framwork spit putput to STDERR by default (as that is
pythons default), but we used to have STDERR, so make it go there again.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We were using the internal imaplib2 _get_untagged_response() functions a
few times. Replace 3 of these calls with 2 calls to the public function
response() rather than fudging with internals that could change anytime.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Remove the old global locking system. We lock only the accounts that we
currently sync, so you can invoke OfflineImap multiple times now as long
as you sync different accounts. This system is compatible with all
releases >= 6.4.0, so don't run older releases simultanous to this one.
This mostly reverts commit 0d95651417,
disabling the old global lock system that we had in parallel to the new one.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The logging rework led to multipline output as we stopped urlencoding
the output lines. Urrg. Fixed this, so output is urlencoded again.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
These were needed for python <2.6 compatability, but since we depend on
python 2.6 now, these can go.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This is a CRITICAL bug fix release for everyone who is on the 6.4.x
series. Please upgrade to avoid potential data loss! The version has
been bumped to 6.5.0, please let everyone know to stay away from 6.5.x!
I am sorry for this.
See details in the Changelog and even more gory details in commit
message for commit 8fc7227189.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This change looks harmless, but it fixes a severe bugfix, potentially
leading to data loss! It fixes the "on n new uploads, it will redownload
n-1, n-2, n-3,... messages during the next syncs" condition, and this is
what happens:
If there are more than one Mails to upload to a server, we do that by
repeatedly invoking folder.IMAP.savemessage(). If the server supports
the UIDPLUS extension we query the resulting UID by doing a:
imapobj._get_untagged_response('APPENDUID', True)
and that is exactly the problem. The "True" part causes the reply to
remain in the "response stack" of the imaplib2 library. When we do
the same call on a subsequent message and the connection is still on the
same folder, we will get the same UID response back (imaplib2 only looks
for the first matching response and returns that). The only time we
clear the response stack, is when the IMAP connection SELECTS a
different folder.
This means that when we upload 10 messages, the IMAP server gives us
always the same UID (that of the first one) back. And trying to write
out 10 different messages with the same UID will confuse OfflineIMAP.
This is the reason why we saw the ongoing UPLOADING/DOWNLOADING behavior
that people reported. And this is the reason why we saw the
inconsistency in the UID mapping in the IMAP<->IMAP case.
I urge everyone to upgrade ASAP. Sorry for that, I don't know why the
problem only became prevalent in the recent few releases as this code
has been there for quite a while.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rather than always parsing the filename, we only need to do so if the flags
have actually changed, otherwise we can keep the filename.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Previously, assigning a new UID to a mapped IMAP or Maildir repository
was done by loading the "local" item, saving it under a new UID and
deleting the old one. This involved lots of disk activity for nothing
more than an effective file rename in Maildirs, and lots of network
usage in the MappedUID cases.
We do this on every upload from a local to a remote item, so that can
potentially be quite expensive. This patch lets backends that support it
(Maildir, MappedUID) efficiently rename the file rather than having to
read the mail content, write it out as a new file and delete the old
file. This speeds up uploads from Maildir and the MappedUID server.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Various functions (such as change_message_uid) will want to construct
maildir filenames, so factor out the code into a helper.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Create a helper function that retrieves the UID, folder MD5, and Flags from
a message filename.
We need these items when we simply want to rename (=new UID) a Maildir
message file later. The new function can give us these components.
Rework, so we cache the calculation of the folder's md5 value once, it
never changes and we call it a lot.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
If someone had a custom :2,a flag, adding a new flag would lead to the
invalid maildir filename ...a:2,... due to regex deficiencies not coping
with this. Fix this so we alway produce valid maildir names.
Note that custom flags are still problematic: as the syncing to the
remote IMAP server will fail, the next sync will assume that they have
been removed from the remote IMAP side and they will be removed from the
local Maildir then. We will need to think about how to handle this. At
least, with this patch we won't lose standard flags and won't produce
invalid maildir names.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The regex for catching Maildir message flags was
self.infosep + '.*2,([A-Z]+)' (infosep being ':').
The .* is bogus, as there is nothing between the : and the 2, per
maildir name specification, so remove that unneeded piece.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
When a new remote folder was detected, we tried to create the folder
locally on the Maildir and called repository.forgetfolders() to force a
new scanning of the Maildir. However, that implementation used the
inherited base function that did nothing. We simply needed to implement
forgetfolders() to set self.folder=None, so we would force a new read in
of the updated local folder structure.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
1) Rename the unintuitive repository.syncfoldersto() to
sync_folder_structure()
2) We were checking if the local repository is readonly and then turning
off any folder creation. But as we can create folders on a remote
repository too, we need to be more fine grained here. Just don't create
a folder on the repository that is marked readonly=True.
This still does not do away with the error message that one currently
gets on missing local folders.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
