To further ensure that messages are synchronized strictly in UID
order, this option can be set to cause only one thread to be used
to synchronise an individual folder, though other folders may
be synchronized simultaneously by other threads.
Signed-off-by: James E. Blair <corvus@gnu.org>
Based-on-patch-by: James E. Blair <corvus@gnu.org>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
OfflineIMAP frequently delivers mail files to the Maildir consisting exclusively
of a single ASCII digit in IDLE mode. IMAPFolder.getmessage expects 'data' to be
of the form
[(fetch-info, message-body)]
However, the imapobj.uid call in getmessage returns a list of *all* pending
untagged FETCH responses. If any message flags were changed in the selected
IMAP folder since the last command (by another client or another thread in
OfflineIMAP itself), the IMAP server will issue unsolicited FETCH responses
indicating these flag changes (RFC3501, section 7). When this happens, 'data'
will look like, for example
['1231 (FLAGS (\\Seen) UID 5300)',
'1238 (FLAGS (\\Seen) UID 5318)',
('1242 (UID 5325 BODY[] {7976}', message-body)]
Unfortunately, getmessage retrieves the message body as data[0][1], which in
this example is just the string "2", and this is what gets stored in the mail
file.
Multi-threaded OfflineIMAP with IDLE or holdconnectionopen is particularly
susceptible to this problem because flag changes synced back to the IMAP server
on one thread will appear as unsolicited FETCH responses on another thread if it
happens to have the same folder selected. This can also happen without IDLE or
holdconnectionopen or even in single-threaded OfflineIMAP with concurrent access
from other IMAP clients (webmail clients, etc.), though the window for the bug
is much smaller.
Ideally, either imaplib2 or getmessage would parse the fetch responses to find
the response for the requested UID. However, since IMAP only specifies
unilateral FETCH responses for flag changes, it's almost certainly safe to
simply find the element of 'data' that is a tuple (perhaps aborting if there is
more than one tuple) and use that.
Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/162
Based-on-patch-by: Austin Clements <amdragon@MIT.EDU>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The virtual imaplib2 first try to import imaplib2 when provided by the system.
If not provided or if the version is not supported, fallback on the bundled
imaplib2 version.
Distribution maintainers can now easily remove the bundled imaplib2 version if
they want to get it packaged outside of offlineimap.
We still want to provide imaplib2 by default because:
- this library is neither in Python core nor packaged by a lot of distributions;
- users expect to be able to run offlineimap by just downloading the tarball or
after a git clone.
In order to avoid unexpected (too old) versions of imaplib2, we restrict the
supported versions of this librabry.
Reviewed-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Some servers are broken: they return zero as valid UID and respond "BAD invalid
parameter: 0" on the FETCH command.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
There is no other way to make Python2 and Python3 happy, because syntax
raise E, V, T is incompatible with the latter.
Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
xrange was removed in Python3 while range exists in boths versions.
Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
"imapobj.search" returns a list with one string element of numbers
separated by one whitespace character for regular box (GMail, AOL...).
['1 2 3 4 5 6 7 8 9 10 11 12']
But if we would like to sync from Davmail it would return a list of
numbers.
['1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12'].
The code "return res_data[0].split()" in the first case will return what
we already have when using Davmail, hence only one email will be
fetched. But if only the first sync would be with maxage the emails
will be removed, because offlineimap will think that they were removed
by us.
The patch distinguishes between syncing with Davmail and regular box and
applies split on the first element only when it finds whitespace
character. It also handles the case when the first element is empty on
first sync.
Closes#327
Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The retry loop would release connection that would get
re-released upon ``finally`` clause. In consequence, an
exception would be cast.
Signed-off-by: Valentin Lab <valentin.lab@kalysto.org>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The keywords are in the flag string, so imaputil can just strip the
usual \Flags.
Signed-off-by: Igor Almeida <igor.contato@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
1. When using maxage, local and remote messagelists are supposed to only
contain messages from at most maxage days ago. But local and remote used
different timezones to calculate what "maxage days ago" means, resulting
in removals on one side. Now, we ask the local folder for maxage days'
worth of mail, find the lowest UID, and then ask the remote folder for
all UID's starting with that lowest one.
2. maxage was fundamentally wrong in the IMAP-IMAP case: it assumed that
remote messages have UIDs in the same order as their local counterparts,
which could be false, e.g. when messages are copied in quick succession.
So, remove support for maxage in the IMAP-IMAP case.
3. Add startdate option for IMAP-IMAP syncs: use messages from the given
repository starting at startdate, and all messages from the other
repository. In the first sync, the other repository must be empty.
4. Allow maxage to be specified either as number of days to sync (as
previously) or as a fixed date.
Signed-off-by: Janna Martl <janna.martl109@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Some messages were excluded from the copy/delete list after the UI message said
they were copied/deleted.
Also fix Internaldate2epoch(), which was wrong.
Signed-off-by: Janna Martl <janna.martl109@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We have no variable "fullname", it must have been slipped in
unintentionally.
Blame commit:
0f40ca4799 more style consistency
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We usually mutate some exceptions to OfflineImapError() and it is
a whole lot better if such exception will show up with the original
traceback, so all valid occurrences of such mutations were transformed
to the 3-tuple form of "raise". Had also added coding guidelines
document where this re-raise strategy is documented.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Create initializer function that puts default values to all fields
of message list item. Fix all code that directly assigns some hash
to the elements of messagelist: for direct assignments only initializer
is now permitted, all other modification are done in-place.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
For servers without UIDPLUS we are inserting additional header
just after transformation '\n' -> CRLF was done. addmessageheaders()
was written to work with just '\n' as the separator, so X-OfflineIMAP
header wasn't preceeded by the CRLF, but just by '\n'.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
When synclabels config flag is set to "yes" for the GMail repo,
offlineimap fetches the message labels along with the messages, and
embeds them into the body under the header X-Keywords (or whatever
'labelsheader' was set to), as a comma separated list.
It also adds an extra pass to savemessageto, that performs label
synchronization on existing messages from GMail to local, the same way
it is done with flags.
We also introduce GmailMaildir repository that adds functionality to
change message labels. It keeps track of messages modification time,
so one can quickly detect when the labels may have changed.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
In preparation for GMail label sync, we had split our some functionality
that will be needed further into their own functions. This also permitted
the code to look more compact and concise.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
When filterheaders is set to a comma-separated list of headers,
OfflineIMAP removes those headers from messages before uploading them
to the server.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Since we just do multiple passes for saving the message without
actually modifying its content (apart from header insertion that
is CRLF-clean), we can change line ends to the proper CRLF just
once.
And we can also get message's date only once too.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Make external API of class/module to be smaller, explicitely mark
all internal functions. Also annotate methods that are implemented
as the part of the parent class interface.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
This commit fixes the case when we're invoking releaseconnection()
for a given imapobj twice.
This bug manifests itself as
{{{
ValueError: list.remove(x): x not in list
File "[...]/offlineimap/folder/IMAP.py", line 615, in savemessage
self.imapserver.releaseconnection(imapobj)
File "[...]/offlineimap/imapserver.py", line 130, in releaseconnection
self.assignedconnections.remove(connection)
}}}
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Global or per-repository option utime_from_message tells OfflineIMAP
to set file modification time of messages pushed from one repository
to another basing on the message's "Date" header.
This is useful if you are doing some processing/finding on your
Maildir (for example, finding messages older than 3 months),
without parsing each file/message content.
From: Cyril RUSSO <boite.pour.spam@gmail.com>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
IMAP servers can return `NO` responses to the `APPEND` command,
e.g. here's an example response from Groupwise's IMAP server:
NO APPEND The 1500 MB storage limit has been exceeded.
In this case, savemessage() should abort the repository sync rather
than returning UID 0 which would cause the local copy of the message
being saved to get irreversibly deleted.
Signed-off-by: Adam Spiers <offlineimap@adamspiers.org>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
They are redundant in all pruned cases and sometimes even create some
problems, e.g., when one tries to jump through paragraphs in vi.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Fix imapfolder.getmessageinternaldate misparsing the Date:
header from emails due to a bug or surprising behaviour by
email.utils.parsedate. This is because email.utils.parsedate's
return value contains the unadjusted hour value from the string
parsed but does not include information about the time zone in
which it is specified. For example (Python 2.7.3):
$ python -c "import email.utils;
print email.utils.parsedate('Mon, 20 Nov 1995 19:12:08 -0500')"
(1995, 11, 20, 19, 12, 8, 0, 1, -1)
(the -1 is the isdst field); the -0500 time zone is completely
ignored, so e.g. the same input with time "19:12:08 +0300" has
the same result. When passed to time.struct_time as allowed per
the parsedate documentation, this time is interpreted in GMT and
thus deviates from the correct value by the timezone offset
(in this example, -5 hours).
I consider this a bug in email.utils.parsedate: In my opinion,
since the return value of the parsetime doesn't include a timezone,
it should be expressed in terms of UTC rather than in terms of the
time zone from the Date header; the existence of
email.utils.parsedate_tz, to which I've switched, indicates that
maybe the authors were aware of this problem.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
