Suppose messages A and B were delivered to the remote folder at
"maxage + 1" days ago.
A was downloaded to the local folder "maxage + 1" days ago, but B was only
downloaded "maxage - 1" days ago (contrived scenario to illustrate the two
things that could happen). The behavior was that B gets deleted from the local
folder, but A did not. The expected behavior is that neither is deleted.
Starting where Base.py: __syncmessagesto_delete(self, dstfolder, statusfolder)
is called where:
- self is the remote folder
and
- dstfolder is the local folder.
It defines deletelist to be the list of messages in the status folder
messagelist that aren't in the remote folder messagelist with
not self.uidexists(uid)
A and B are both in the status folder. They're also both *NOT* in the remote
folder messagelist: this list is formed in IMAP.py: cachemessagelist(), which
calls _msgs_to_fetch(), which only asks the IMAP server for messages that are
"< maxage" days old.
Back to Base.py __syncmessagesto_delete(), look at the call
folder.deletemessages(deletelist), where folder is the local folder. This ends
up calling Maildir.py deletemessage() for each message on the deletelist. But we
see that this methods returns (instead of deleting anything) if the message is
in the local folder's messagelist. This messagelist was created by Maildir.py's
cachemessagelist(), which calls _scanfolder(), which tries to exclude messages
based on maxage. So at this point, we *WANT* A and B to be excluded -- then they
will be spared from deletion. This maxage check calls _iswithinmaxage(), and
actually does the date comparison based on the time found at the beginning of
the message's filename. These filenames were originally created in Maildir.py's
new_message_filename(), which calls _gettimeseq() to get the current time (i.e.
the time of retrieval).
Upshot: A's filename has an older timestamp than B's filename. A is excluded
from the local folder messagelist in _scanfolder(), hence spared from deletion
in deletemessage(); B is not excluded, and is deleted.
This patch does not address the timezone issue. As for the IMAP/timezone issue,
a similar issue is discussed in the thunderbird bug tracker here:
https://bugzilla.mozilla.org/show_bug.cgi?id=886534
In the end, they're solving a different problem, but they agree that
there is really no reliable way of guessing the IMAP server's internal
timezone.
Signed-off-by: Janna Martl <janna.martl109@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Read proxy option in imapserver, instantiate a class in imaplibutil
using a self-defined keyword and a socket instance, and use this socket
instance to substitute the default socket instance used in imaplib2.
Signed-off-by: 夏恺(Xia Kai) <xiaket@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Get real tracebacks (at the point that calls Request.abort()
and not from the point that handles collected abort requests)
and pass them to our calling functions to ease debugging
of user problems.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
This simplifies logics for the user, especially if he uses both
fingerprint and certificate validation: it is hard to maintain
the compatibility with the prior behaviour and to avoid getting
default CA bundle to be disabled when fingerprint verification
is requested.
See
http://thread.gmane.org/gmane.mail.imap.offlineimap.general/6695
for discussion about this change.
Default CA bundle is requested via 'sslcertfile = OS-DEFAULT'.
I had also enforced all cases where explicitely-requested CA bundles
are non-existent to be hard errors: when users asks us to use CA
bundle (and, thus, certificate validation), but we can't find one,
we must error out rather than happily continue and downgrade to
no validation.
Reported-By: Edd Barrett <edd@theunixzoo.co.uk>
Reviewed-By: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
We have no variable "fullname", it must have been slipped in
unintentionally.
Blame commit:
0f40ca4799 more style consistency
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The note tells people to look at the source of the method, which
spinx.ext.viewcode conveniently links right next to the methods
signature.
Signed-off-by: Wieland Hoffmann <themineo@gmail.com>
We usually mutate some exceptions to OfflineImapError() and it is
a whole lot better if such exception will show up with the original
traceback, so all valid occurrences of such mutations were transformed
to the 3-tuple form of "raise". Had also added coding guidelines
document where this re-raise strategy is documented.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
CustomConfigParser.getlocaleval() loads "pythonfile" at each call.
Besides unnecessary IO, in case that dynamic_folderfilter is true, the
code in "pythonfile" would behave stateless, since it is re-initialized
at each call of getlocaleval(), i.e., at every sync. Fix that by keeping
a singleton copy of localeval in CustomConfigParser after the first call
of getlocaleval().
Signed-off-by: Stefan Huber <shuber@sthu.org>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Otherwise, messages logged through UIBase.error would only be passed to
UIBase._msg, which only logs at INFO. This causes error to not get
logged at all for the quit UI.
Signed-off-by: Wieland Hoffmann <themineo@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This fixes a bug in which a message ended up with multiple gmail labels
header (X-Keywords or so). Fix fix it by removing all labels headers
before adding the updated one.
Signed-off-by: Abdo Roig-Maranges <abdo.roig@gmail.com>
There should be just one header storing gmail labels, but due to a bug,
multiple X-Keywords (or equivalent) headers may be found on the local
messages.
Now we, when extracting the labels from a message, we read all label
headers, instead of just the first one.
This has the consequence that some old labels stored locally in a second
X-Keywords (or third...) header, which effectively was rendered
invisible to offlineimap until now, may pop back up again and be pushed
to gmail. No labels will be removed by the changes in this commit,
though.
Signed-off-by: Abdo Roig-Maranges <abdo.roig@gmail.com>
It is a bit cleaner than making chains of calls like
{{{
value = os.path.expanduser(value)
value = os.path.abspath(value)
}}}
since we do see all transformations to be applied in a single
iterable and have no repeated code like in the above example.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
- Multi-line documentation for functions and methods
now has ending triple-double-quotes on an own line,
as per PEP 257.
- Added documentation and comments to almost all functions
and methods.
- Added stub implementations for getconfig() and getsection()
inside CustomConfig.ConfigHelperMixin to provide sane
run-time diagnostics for classes that doesn't implement them.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
The GNU GPL and the OpenSSL license are incompatible. Some distributions
take a hardline stance and do not consider OpenSSL to be a systems library
(which would permit the usage/distribution of OpenSSL) when distributing
apps such as OfflineImap from the same repository. In order to solve these
distributions dilemma, we add the OpenSSL exception to our GNU GPL v2+ license.
This allows for unambiguous use/distribution of our GNU GPL'ed application
with a python linking to openssl.
Consent of all contributors has been requested via email by
Sebastian@SSpaeth.de. With very few exceptions of minor contributions (which
might or might not by copyright-worthy) all past contributors have consented
to adding the OpenSSL exception. None of the replying authors has disagreed
with adding the exception.
The corresponding issues at question:
https://github.com/OfflineIMAP/offlineimap/issues/104
Debian bug #747033
We are still missing consent from:
1 Asheesh Laroia
2 Bart Kerkvliet
4 Daniel Burrows
5 David Favro
1 David Logie
1 Eric Dorland
1 Ethan Schoonover
49 Eygene Ryabinkin
1 Loui Chang
1 Luca Capello
1 Michael Witten
2 Mike Dawson
1 Peter Colberg
1 Scott Henson
1 Tom Lawton
1 W. Trevor King
2 X-Ryl669
1 buergi
2 dtk
5 mj
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Commit 7df765cfdb introduced regression:
GmailMaildir caches labels in its own function and it was testing the
presence of the 'labels' key in message descriptor. But 7df765cf
changed descriptor initialization and this key is always present.
So now we have 'labels_cached' flag that tells us if labels were
already cached or not.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Create initializer function that puts default values to all fields
of message list item. Fix all code that directly assigns some hash
to the elements of messagelist: for direct assignments only initializer
is now permitted, all other modification are done in-place.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
The example illustrations were slightly cryptic;
modify them to be more obvious.
Fix case #2 (message starts with two line breaks)
where additional line break was shown and coded.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
The parameter's value is a string representing the linebreak,
and can sometimes contain just '\n', in which case naming it
crlf is slightly misleading.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
For cases like
http://article.gmane.org/gmane.mail.imap.offlineimap.general/6468
it is beneficial to see that folder name was translated and the result
of this translation on a single line: having log like
{{{
Folder Boring/Wreck [acc: tmarble@info9.net]:
Syncing Boring/Breck: Gmail -> Maildir
}}}
with translated name on the "Folder" line and original one on the
"Syncing" line isn't very intuitive.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
It is always good to see which version we're talking about, so I had
added explicit marker for -devel, -release, -rcX and other states of
the OfflineIMAP.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
For servers without UIDPLUS we are inserting additional header
just after transformation '\n' -> CRLF was done. addmessageheaders()
was written to work with just '\n' as the separator, so X-OfflineIMAP
header wasn't preceeded by the CRLF, but just by '\n'.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
private methods prevent them from being overriden on derived classes. In
GmailFolder we need to override copymessageto, so it can't be private.
Before this commit, copymessageto was made private in Base but not in
GmailFolder. The end result was that labels were not set when copying
the message content, and always needed to be set on the label copying
pass.
Pointyhat-to: Eygene Ryabinkin
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
... and not self.filterheaders. With the current code this change
is no-op (since self.filterheaders is always passed as header_list),
but it is a bug in general.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
If when we request a LocalStatus folder, the folder has to be created,
we look whether the other backend has data, and if it does we migrate
it to the new backend.
The old backend data is left untouched, so that if you change back say
from sqlite to plaintext, the older data is still there. That should
not lead to data loss, only a slower sync while the status folder gets
updated.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
* Do not inherit LocalStatusSQLiteFolder class from the plaintext
one.
* Use some functions already in BaseFolder in both, plaintext and
sqlite classes.
* Add a saveall method. The idea is that saveall dumps the entire
messagelist to disk, while save only commits the uncommited
changes. Right now, save is noop for sqlite, and equivalent to
saveall for plaintext, but it enables to be more clever on when we
commit to disk in the future.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
* Implements Status Folder format v2, with a mechanism to upgrade an
old statusfolder.
* Do not warn about Gmail and GmailMaildir needing sqlite backend
anymore.
* Clean repository.LocalStatus reusing some code from
folder.LocalStatus.
* Change field separator in the plaintext file from ':' to '|'. Now
the local status stores gmail labels. If they contain field
separator character (formerly ':'), they get messed up. The new
character '|' is less likely to appear in a label.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Format headers X-Label and Keywords as a space separated list and all
other ones as comma-separated entities. This makes OfflineIMAP label
handling to be compatible with some user agents that recognise these
headers.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
When synclabels config flag is set to "yes" for the GMail repo,
offlineimap fetches the message labels along with the messages, and
embeds them into the body under the header X-Keywords (or whatever
'labelsheader' was set to), as a comma separated list.
It also adds an extra pass to savemessageto, that performs label
synchronization on existing messages from GMail to local, the same way
it is done with flags.
We also introduce GmailMaildir repository that adds functionality to
change message labels. It keeps track of messages modification time,
so one can quickly detect when the labels may have changed.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
In preparation for GMail label sync, we had split our some functionality
that will be needed further into their own functions. This also permitted
the code to look more compact and concise.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
When filterheaders is set to a comma-separated list of headers,
OfflineIMAP removes those headers from messages before uploading them
to the server.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Since we just do multiple passes for saving the message without
actually modifying its content (apart from header insertion that
is CRLF-clean), we can change line ends to the proper CRLF just
once.
And we can also get message's date only once too.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Added the configuration setting usecompression for the IMAP repositories.
When enabled, the data from and to the IMAP server is compressed.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Make external API of class/module to be smaller, explicitely mark
all internal functions. Also annotate methods that are implemented
as the part of the parent class interface.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Allow people who want folder filtering to depend on the external
conditions or to make it dynamic for other reasons, to do what
they want.
New repository configuration knob 'dynamic_folderfilter' was
introduced; it defaults to 'False' that matches historical behaviour.
GitHub: #73
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
This commit fixes the case when we're invoking releaseconnection()
for a given imapobj twice.
This bug manifests itself as
{{{
ValueError: list.remove(x): x not in list
File "[...]/offlineimap/folder/IMAP.py", line 615, in savemessage
self.imapserver.releaseconnection(imapobj)
File "[...]/offlineimap/imapserver.py", line 130, in releaseconnection
self.assignedconnections.remove(connection)
}}}
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
When imaputil was parsing quoted strings, it treated "abcd\\"
as incomplete quoted string having escaped quote, rather than
properly-quoted string having escaped backslash.
GitHub issue: https://github.com/OfflineIMAP/offlineimap/issues/53
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2.36 it includes support for SSL version override that was integrated
into our code before, no other changes.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
- Fix for Gmail sending a BYE response after reading >100 messages
in a session.
- Includes fix for GitHub#15: patch was accepted upstream.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Second argument is exception traceback, not the message; without this
tracebacks like mentioned in
http://permalink.gmane.org/gmane.mail.imap.offlineimap.general/5712
were happening when this exception handling block was hit.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
This greatly simplifies developer's life and will, possibly, allow
users familiar with Python to debug and fix the problems by
themselves.
We, possibly, should not give tracebacks for the problems like
"can't open connection", but this is up to the caller of this
routine not to provide traceback in this case.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
self._expecting_data was used both as the expected data length and the
flag that we expect some data. This obviously fails when advertized
data length is zero, so self._expecting_data_len was introduced to
hold the length of the expected data and self._expecting_data was left
as the flag that we expect the data to come.
GitHub issue: https://github.com/OfflineIMAP/offlineimap/issues/15
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
It is called localfolders and holds expanded name for the same
variable for the local repository of the account that is being
processed.
GitHub issue: https://github.com/OfflineIMAP/offlineimap/issues/21
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Bring the description in the template offlineimap.conf in sync to the
actual implementation: pass folder names to the sorting function, not
the offlineimap.folder.IMAP.IMAPFolder objects themselves.
GitHub issue: https://github.com/OfflineIMAP/offlineimap/issues/27
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Global or per-repository option utime_from_message tells OfflineIMAP
to set file modification time of messages pushed from one repository
to another basing on the message's "Date" header.
This is useful if you are doing some processing/finding on your
Maildir (for example, finding messages older than 3 months),
without parsing each file/message content.
From: Cyril RUSSO <boite.pour.spam@gmail.com>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Added configuration option "auth_mechanisms" to the config file:
it is a list of mechanisms that will be tried in the specified order.
Author: Andreas Mack <andreas.mack@konsec.com>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
It's nice to set up an ssh tunnel command which forwards an IMAP tcp
port inside an encrypted session, e.g. with ssh's "-W" flag. In this
case the tunnelled connection still requires authentication inside
IMAP session, because this is transport-only tunnel that substitutes
normal TCP/SSL connection.
New directive, 'transporttunnel' was added: it specifies the command
that will create the tunnel. Only one type of tunnel must be
specified for a single repository: we can't have both preauthenticated
and transport-type tunnels, they won't chain together.
From: Steve Purcell <steve@sanityinc.com>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
- this method isn't as deprecated as IMAP LOGIN;
- it allows to keep hashed passwords on the server side;
- it has the ability to specify that the remote identity
is different from authenticating username, so it even
can be useful in some cases (e.g., migrated mailboxes);
configuration variable "remote_identity" was introduced
to leverage this functionality.
From: Andreas Mack <andreas.mack@konsec.com>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
- created helper routine that will do authentication;
- routine tries each method in turn, first successful
one terminates it: makes things easier to read
and handle;
- renamed plainauth() inside offlineimap/imapserver.py
to loginauth(): the function does IMAP LOGIN authentication
and there is PLAIN SASL method, so previous name was
a bit misleading;
- slightly improved error reporting: all exceptions during
authentication will be reported at the end of the run;
- now loginauth() is never called if LOGINDISABLED is advertized
by the server; it used to be invoked unconditionally when
CRAM-MD5 fails, but we should respect server's opinion on
how to handle its users.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
IMAP servers can return `NO` responses to the `APPEND` command,
e.g. here's an example response from Groupwise's IMAP server:
NO APPEND The 1500 MB storage limit has been exceeded.
In this case, savemessage() should abort the repository sync rather
than returning UID 0 which would cause the local copy of the message
being saved to get irreversibly deleted.
Signed-off-by: Adam Spiers <offlineimap@adamspiers.org>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
mutt-sidebar and, probably, other MUA show mailboxes in the order
they are listed in the file written by mbnames. Therefore, to allow
customization of the order with which mailboxes are listed, introduce
the new 'sort_keyfunc' directive in the [mbnames] section.
'sort_keyfunc' must be a function that will be called once for each
mailbox. It must accept the only argument -- a dict with 2 items,
'accountname' and 'foldername', and should return an object that
will be used as the sorting key for each mailbox.
Default key function returns (d['accountname'], d['foldername']),
thus sorting by account name and then by the folder name.
Signed-off-by: Johan Herland <johan@herland.net>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Make IDLE syncs be equal to the regular synchronisations
in respect to pre-sync and post-sync hooks.
From: mxgr7 <maxgerer@gmail.com>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Run the locked code under 'with': this guarantees that lock
will be released in any case.
This modification also avoids the case when our thread wasn't running
locked when exception was caught, another thread got the lock, our
code checked it via self.connectionlock.locked() and errorneously
released the lock thinking that is was running locked.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
They are redundant in all pruned cases and sometimes even create some
problems, e.g., when one tries to jump through paragraphs in vi.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
We now allow setting the SSL version used when connecting to IMAPS servers, and
do so via the `ssl_version` configuration option. We default to the current
practice (letting python's "ssl" library automatically detect the correct
version). There are however rare cases where one must specify the version to
use.
Signed-off-by: Ryan Kavanagh <rak@debian.org>
This eases testing of option values inside the code. This instance
is implemented as the read-only copy of the obtained 'options' object,
so callers won't be able to modify its contents.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
The problem lies in the fact that offlineimap.folder.Base's method
syncmessagesto_copy() uses threaded code everytime it is suggested by
the derived class's suggeststhreads() (currently, only IMAP does this
suggestion), but offlineimap/init.py will not spawn the
exitnotifymonitorloop() from offlineimap.threadutil.
The root cause is that ExitNotifyThread-derived threads need
offlineimap.threadutil's exitnotifymonitorloop() to be running the
cleaner for the exitthreads Queue(), because it fills the queue via
the run() method from this class: it wants to put() itself to the
Queue on exit, so when no exitnotifymonitorloop() is running, the
queue will fill up. And if this thread is an instance of
InstanceLimitedThread that hits the limit on the number of threads,
then it will hold the instancelimitedsems[] semaphore will prevent
other InstanceLimitedThread()s of the same name to pass its start()
method.
The fix is to avoid using threaded code if we're running
single-threaded.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Obtained-from: X-Ryl669 <boite.pour.spam@gmail.com>
