Otherwise, it might be impossible to know which account is connecting when more
than one is syncing.
Code style.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
They introduce a regression not allowing to discard the XOAUTH2 method when
expected.
The default lambda did not take the "account_name" argument.
Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/362
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
OfflineIMAP frequently delivers mail files to the Maildir consisting exclusively
of a single ASCII digit in IDLE mode. IMAPFolder.getmessage expects 'data' to be
of the form
[(fetch-info, message-body)]
However, the imapobj.uid call in getmessage returns a list of *all* pending
untagged FETCH responses. If any message flags were changed in the selected
IMAP folder since the last command (by another client or another thread in
OfflineIMAP itself), the IMAP server will issue unsolicited FETCH responses
indicating these flag changes (RFC3501, section 7). When this happens, 'data'
will look like, for example
['1231 (FLAGS (\\Seen) UID 5300)',
'1238 (FLAGS (\\Seen) UID 5318)',
('1242 (UID 5325 BODY[] {7976}', message-body)]
Unfortunately, getmessage retrieves the message body as data[0][1], which in
this example is just the string "2", and this is what gets stored in the mail
file.
Multi-threaded OfflineIMAP with IDLE or holdconnectionopen is particularly
susceptible to this problem because flag changes synced back to the IMAP server
on one thread will appear as unsolicited FETCH responses on another thread if it
happens to have the same folder selected. This can also happen without IDLE or
holdconnectionopen or even in single-threaded OfflineIMAP with concurrent access
from other IMAP clients (webmail clients, etc.), though the window for the bug
is much smaller.
Ideally, either imaplib2 or getmessage would parse the fetch responses to find
the response for the requested UID. However, since IMAP only specifies
unilateral FETCH responses for flag changes, it's almost certainly safe to
simply find the element of 'data' that is a tuple (perhaps aborting if there is
more than one tuple) and use that.
Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/162
Based-on-patch-by: Austin Clements <amdragon@MIT.EDU>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Commit 6b28071b0f makes the check once nametrans is applied. This is wrong
because separators are valid characters in names returned by nametrans. E.g:
"Sent" -> "[Gmail]/Sent"
Make the check against the raw name of the folder, instead.
Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/353
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
1. There is one database per folder and sqlite requires to serialize the
writings. Instead of locking at LocalStatusSQLiteFolder object level, introduce
a new DatabaseFileLock object which is shared across threads. This fixes the
concurrent writes issues that some users might experience by duplications or
flags restored to the previous state.
2. Close the database only when we are sure no other threads will use the
connection on a *per-file* basis. Previous fix 677afb8d8f is wrong
because the same lock is shared for all the databases.
Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/350
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
1. There is one database per folder and sqlite requires to serialize the
writings. Instead of locking at LocalStatusSQLiteFolder instance level,
introduce a new DatabaseFileLock object which is shared across threads. This
fixes the concurrent writes issues that some users might experience by
duplications or flags restored to the previous state.
2. Close the database only when we are sure no other threads will use the
connection on a *per-file* basis. Previous fix 677afb8d8f is wrong
because the same lock is shared for all the database files.
Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/350
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This did not work and is not usefull since the purge() method was introduced in
1410a391bc. Actually, the purge() does what deletemessagelist() was supposed to
achieve.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
It's required to have more than one connection to the database for the
maxconnections configuration option to work with threads. However,
connection.close() is closing all the connections. Only close the connection
when no more thread need it.
Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/350
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
It's required to have more than one connection to the database for the
maxconnections configuration option to work with threads. However,
connection.close() is closing all the connections. Only close the connection
when no more thread need it.
Backported-from: 856b74407bd7f634cae5a8c2d9b84e13d14c12d2
Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/350
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This is usefull to remove dangling entries for removed accounts or if mbnames is
not enabled anymore.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Output a warning so that we can dig into this. For now, the patch doesn't fix
the root cause. If the server returns UID 0 as valid UID number, this must be
ignored as soon as possible.
Github-ref: https://github.com/OfflineIMAP/offlineimap/issues/336
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
When a maildir is removed it must be considered new for the sync. However, the
local cache of the folder remains. This means the sync of the folder removes all
the missing emails.
Avoid loosing of data for users not aware of the local cache by removing any
pre-existing status cache of a folder when we actually want to create the
database.
Improve style.
Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/333
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
- Learn to support UTF-8 characters where it was not supported for usernames and
passwords (but for netrc).
- Fix the types in the code for both py2 and py3: we now expect unicode for
usernames and passwords.
Unicode (UTF-8) is required only for variables with non-ASCII characters.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Make mbnames to work with intermediate files, one per account, in the JSON
format. The mbnames target is built from those intermediate files.
Github-Fix: https://github.com/OfflineIMAP/offlineimap/issues/66
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
- Expose more literals.
- Fix a type to bytes.
- Fix comment.
Changes submitted upstream by Łukasz and me.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Backported from 8e995a69bfa003ab822b55731429d84b3bc5626f.
We currently close the database as soon as possible while we handle the status
backend but this is still too early because autorefresh induces looping on this
code block.
Instead of delaying the closing outside of the loop, it's easier to delay the
opening as late as possible (inside the loop). The downside is that the database
is opened/closed more than once when autorefresh is enabled. The good news is
that this make the code much easier.
Fixes regression introduces by 6fb5700.
Reported-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The virtual imaplib2 first try to import imaplib2 when provided by the system.
If not provided or if the version is not supported, fallback on the bundled
imaplib2 version.
Distribution maintainers can now easily remove the bundled imaplib2 version if
they want to get it packaged outside of offlineimap.
We still want to provide imaplib2 by default because:
- this library is neither in Python core nor packaged by a lot of distributions;
- users expect to be able to run offlineimap by just downloading the tarball or
after a git clone.
In order to avoid unexpected (too old) versions of imaplib2, we restrict the
supported versions of this librabry.
Reviewed-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Be aware upstream has two versions of both 2.52 and 2.53.
This bump change is from
commit a205409a0047732840505e534c07d8d85d2644a1
Author: Piers Lauder <piers@janeelix.com>
Date: Sun Dec 27 20:55:04 2015 +1100
Back out BINARY changes - not implemented correctly.
to
commit 0596e7372fd3556d27ea55510b1e8cfa8370ec43
Author: Piers Lauder <piers@janeelix.com>
Date: Fri Jun 3 11:58:40 2016 +1000
new version of imaplib2.py
Original-imaplib2-patch-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This patch fixes the situation when "localfolders" specifies path that
is more that one level deep and top directory does not exists. Example
would be "localfolders = ~/Mail/a". This especially relevant on the
first run.
In that case we would end up with unhandled exception causing
unexpected termination of the program.
Thread 'Account sync test' terminated with exception:
Traceback (most recent call last):
File "/offlineimap/offlineimap/threadutil.py", line 172, in run
Thread.run(self)
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/offlineimap/offlineimap/accounts.py", line 258, in syncrunner
self.localrepos = Repository(self, 'local')
File "/offlineimap/offlineimap/repository/__init__.py", line 82, in __new__
return repo(name, account)
File "/offlineimap/offlineimap/repository/Maildir.py", line 40, in __init__
os.mkdir(self.root, 0o700)
OSError: [Errno 2] No such file or directory: '/Mail/a'
By replacing call to "mkdir" with "makedirs" we can simply create
directories recursively.
Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Some servers are broken: they return zero as valid UID and respond "BAD invalid
parameter: 0" on the FETCH command.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
There is no other way to make Python2 and Python3 happy, because syntax
raise E, V, T is incompatible with the latter.
Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Python3 accepts binary input for md5 function.
This patch is known to break setups using folder names not strictly conforming
the IMAP UTF-7 encoding. We always made it clear that such setup is unsupported
and might be broken at some point in time. See documentation about
'decodefoldernames' in the provided configuration file. This is why this patch
is considered introducing no regression for this use case.
Patches to support both Python 3 and Python 2 by re-encoding the MD5 in the
filenames are welcome. This likely requires a new CLI option to allow
backporting the feature for users downgrading or changing of Python environment.
Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We currently close the database as soon as possible while we handle the status
backend but this is still too early because autorefresh induces looping on this
code block.
Instead of delaying the closing outside of the loop, it's easier to delay the
opening as late as possible (inside the loop). The downside is that the database
is opened/closed more than once when autorefresh is enabled. The good news is
that this make the code much easier.
Fixes regression introduces by 6fb5700.
Reported-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
With a maxsize to the Queue of threads we are introducing a blocking call while
adding new threads.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
In Python3, zip returns iterator instead of list.
Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
xrange was removed in Python3 while range exists in boths versions.
Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
There is no such method in Python3 any more.
Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
It is more readable and returns a list therefore it is compatible both
with Python 2 and 3.
Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
"imapobj.search" returns a list with one string element of numbers
separated by one whitespace character for regular box (GMail, AOL...).
['1 2 3 4 5 6 7 8 9 10 11 12']
But if we would like to sync from Davmail it would return a list of
numbers.
['1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12'].
The code "return res_data[0].split()" in the first case will return what
we already have when using Davmail, hence only one email will be
fetched. But if only the first sync would be with maxage the emails
will be removed, because offlineimap will think that they were removed
by us.
The patch distinguishes between syncing with Davmail and regular box and
applies split on the first element only when it finds whitespace
character. It also handles the case when the first element is empty on
first sync.
Closes#327
Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Sometimes it might happen that you put wrong date and you except emails
to be fetched, but they are not and you do not have an idea why.
By raising exception the user will see a proper message telling that he
used the wrong date for maxage/startdate property.
If someone wants to set a future date intentionally might as well sync
in the future.
Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This enables the "append" mode feature. Configuration option is sync_deletes in
both local and remote repositories. Marked EXPERIMENTAL and UNTESTED.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Introduce the '--migrate-fmd5-using-nametrans' option which migrates the
FMD5 hashes from versions prior to 6.3.5.
It seems that commit 'Apply nametrans to all Foldertypes' (6b2ec956cf)
introduced a regression because it changed the FMD5 part of the filename
calculated by OfflineIMAP. Thus, OfflineIMAP believes that the messages
has been removed and adds them back.
For more information, see:
http://www.offlineimap.org/configuration/2016/02/12/debian-upgrade-from-jessie-to-stretch.html
Bug-Debian: https://bugs.debian.org/812108
Reported-by: François <francois@avalenn.eu>
Signed-off-by: Ilias Tsitsimpis <i.tsitsimpis@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Some environments that return AAAA records for their IMAP servers can pose
problems for clients that do not have end-to-end IPv6 connectivity for a number
of reasons (e.g. policy, lack of full routing, security, etc..)
Even with a fallback mechanism in place, you can still arrive at IMAP
implementations that could prevent authentication from unknown IPv6 space. This
in itself is not enough to fallback to IPv4 since there is an actual connection
on that socket.
This change is for introducing a user-defined value:
[Repository imap-remote]
ipv6 = no
to create a preference per repository on which AF to connect to the remote
server on
ipv6 = yes (AF_INET6)
ipv6 = no (AF_INET)
unspecified = default
Signed-off-by: Ebben Aries <e@dscp.org>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Since this is used in an except calse, we first don't mask the real cause and
raise the original error.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
By default, OfflineIMAP catches SIGTERM/SIGHUP/SIGINT and attempts to
gracefully terminate as soon as possible. Allow the user to abort
immediately, by hitting Ctrl-C several times.
Bug-Debian: https://bugs.debian.org/679975
Signed-off-by: Ilias Tsitsimpis <i.tsitsimpis@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This reverts commit a1dc76ae91.
Causes a crash when using Blinkenlights UI with -l CLI option.
$ ./offlineimap.py -c offlineimap.conf.minimal -u blinkenlights -l foo
Traceback (most recent call last):
File "./offlineimap.py", line 36, in <module>
oi.run()
File "/tmp/offlineimap/offlineimap/init.py", line 50, in run
options, args = self.__parse_cmd_options()
File "/tmp/offlineimap/offlineimap/init.py", line 205, in __parse_cmd_options
self.ui.setlogfile(options.logfile)
File "/tmp/offlineimap/offlineimap/ui/UIBase.py", line 119, in setlogfile
self.logger.info(msg)
File "/usr/lib/python2.7/logging/__init__.py", line 1159, in info
self._log(INFO, msg, args, **kwargs)
File "/usr/lib/python2.7/logging/__init__.py", line 1278, in _log
self.handle(record)
File "/usr/lib/python2.7/logging/__init__.py", line 1288, in handle
self.callHandlers(record)
File "/usr/lib/python2.7/logging/__init__.py", line 1328, in callHandlers
hdlr.handle(record)
File "/usr/lib/python2.7/logging/__init__.py", line 751, in handle
self.emit(record)
File "/tmp/offlineimap/offlineimap/ui/Curses.py", line 305, in emit
color = self.ui.gettf().curses_color
AttributeError: 'CursesLogHandler' object has no attribute 'ui'
Reported-by: iliastsi
Github-issue: #293
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The retry loop would release connection that would get
re-released upon ``finally`` clause. In consequence, an
exception would be cast.
Signed-off-by: Valentin Lab <valentin.lab@kalysto.org>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This fix does not apply when any keyword in configured which is already
harmless.
Written-by: Igor Almeida <igor.contato@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Extract the flag/keyword translation and combination logic to a
function.
Signed-off-by: Igor Almeida <igor.contato@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This uses the destination folder's keyword mapping to translate the
message's keywords into some appropriate format.
Tested only with local Maildir.
Signed-off-by: Igor Almeida <igor.contato@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The keywords are in the flag string, so imaputil can just strip the
usual \Flags.
Signed-off-by: Igor Almeida <igor.contato@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This commit assembles a dictionary mapping user-specified IMAP keywords
to Maildir lower-case flags, similar to Dovecot's format
http://wiki2.dovecot.org/MailboxFormat/Maildir
Configuration example:
[Repository Local]
type = Maildir
localfolders = ~/Maildir/
customflag_a = $label1
customflag_b = $Forwarded
customflag_c = Junk
Signed-off-by: Igor Almeida <igor.contato@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Remove filtering that was previously done to avoid errors in flag
handling.
Signed-off-by: Igor Almeida <igor.contato@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Note that the stacks are grouped if similar, and the current
process (the one handling the signal) is identified and reports
where it was before the signal.
This can be quite handy when wanting to debug thread locks for
instance.
Signed-off-by: Valentin Lab <valentin.lab@kalysto.org>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
If this value is true, use (if possible) a timestamp based on message
Date or Delivery-date headers. The current system time is used
otherwise.
filename_use_mail_timestamp and utime_from_header are now completely
separated option that do not interfere one with other.
To handle this feature in a multithread context we use a hash to count
the number of mail with the same timestamp. This method is more accurate
than using the old lasttime and timeseq variables.
Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Rather than having an option for syslog output, make a separate UI
option.
Signed-off-by: Ben Boeckel <mathstuf@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Handle case where email's internal time is erroneously so large as to
cause overflow errors when setting file modification time with
utime_from_header = true.
Reported-by: Cameron Simpson <cs@zip.com.au>
Signed-off-by: Janna Martl <janna.martl109@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Allow the user to block usage of known-bad versions of SSL and TLS.
Signed-off-by: Ben Boeckel <mathstuf@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Return value set to 1 if sync failed and error is thrown. Otherwise set to 0 if successful.
Signed-off-by: Prashant Sachdeva <sachdevp@cs.ubc.ca>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
When new mail arrives, this hook is triggered, allowing the user to
play a sound, or launch a popup.
Signed-off-by: Matthew Krafczyk <krafczyk.matthew@gmail.com>
Since skipping a folder means no new data is downloaded, the UID validity
problem is a backup failure. Make it possible to alert or work around
it in scripts by signaling with the exit code.
Signed-off-by: Alex Kapranoff <alex@kapranoff.ru>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Now it is possible to handle failed syncs in scripts.
Signed-off-by: Alex Kapranoff <alex@kapranoff.ru>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
1. When using maxage, local and remote messagelists are supposed to only
contain messages from at most maxage days ago. But local and remote used
different timezones to calculate what "maxage days ago" means, resulting
in removals on one side. Now, we ask the local folder for maxage days'
worth of mail, find the lowest UID, and then ask the remote folder for
all UID's starting with that lowest one.
2. maxage was fundamentally wrong in the IMAP-IMAP case: it assumed that
remote messages have UIDs in the same order as their local counterparts,
which could be false, e.g. when messages are copied in quick succession.
So, remove support for maxage in the IMAP-IMAP case.
3. Add startdate option for IMAP-IMAP syncs: use messages from the given
repository starting at startdate, and all messages from the other
repository. In the first sync, the other repository must be empty.
4. Allow maxage to be specified either as number of days to sync (as
previously) or as a fixed date.
Signed-off-by: Janna Martl <janna.martl109@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
