Allow retrying the download of messages more than twice.
Signed-off-by: Luke Kenneth Casson Leighton <lkcl@lkcl.net>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Allow to use a another proxy for authentication than for IMAP.
Signed-off-by: Luke Kenneth Casson Leighton <lkcl@lkcl.net>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Commit e51ed80e claims to add tilde and environment variable expansion
to multiple locations including Repository.localfolders. However, this
particular options seems to have been missed in that commit, and
apparently no one noticed till date
Signed-off-by: Darshit Shah <darnir@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
481efa95f6 is wrong to assume the local nametrans is set up and working.
Now, we correctly check that the local folder name is the same once both
nametrans are applied to itself. This check is only done when we have to create
the folder on remote.
Github-ref: https://github.com/OfflineIMAP/offlineimap/issues/413
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
When creating a folder on the remote, the status folder was created
using the remote folder name, but then Account assumed the statusfolder
name matched the local folder. Correct this.
Signed-off-by: Abdo Roig-Maranges <abdo.roig@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Ensure we work on the correct names when coparing the structures.
This might revert changes made in 22641331c1 and would require mode
checks. However, having correct folder structure comparison is more important
than having the not official UTF-8 support.
Github-ref: https://github.com/OfflineIMAP/offlineimap/issues/405
Tested-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The GMail repository added in 81b86fb has the restriction that you
can't override the host/port/ssl etc. There's no good reason for this,
and I'd like to use some of the defaults while connecting to GMail via
an stunnel, i.e.:
$ grep -v -e ^# -e ^$ .stunnel.work-gmail-imap
foreground = yes
debug = info
pid =
[work-imap-gmail-ssl-tunnel]
client = yes
accept = 127.0.0.1:1431
connect = imap.gmail.com:993
delay = yes
And then in my .offlineimaprc:
type = Gmail
remotehost = localhost
remoteport = 1431
ssl = no
While I was at it I also allowed overriding all the other previously
hardcoded options, e.g. the "oauth2_request_url" can now be overridden
to e.g. a local non-https proxy to sniff & debug the traffic going
over the wire.
This doesn't break any existing configuration, since we still use the
provided defaults. It just allows odd uses like my own to use this
repository while e.g. not having to supply "trashfolder", "spamfolder"
& "oauth2_request_url". While I'm at it document in the
offlineimap.conf that we're setting those options to defaults in this
repository.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
get_os_name returns linux-opensuse on OpenSUSE, so add a line for linux-opensuse to __DEF_OS_LOCATIONS.
Signed-off-by: Michael Hohmuth <hohmuth@sax.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
sync_folder_structure wasn't applying utf7 decoding before comparing
folders.
This is a minimal fix that only modifies one function so it doesn't
interfere with rest of the code.
Signed-off-by: Stéphane Albert <sheeprine@oh.its.fake.nullplace.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Using standard offlineimap without specific utf8 nametrans makes
offlineimap crash when generating md5 of the folder because the
foldername is already an str.
Signed-off-by: Stéphane Albert <sheeprine@oh.its.fake.nullplace.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This should significantly improve performance when used to write large
amounts of messages.
This feature is enabled through the fsync configuration option.
Code refactorize around fsync.
This addresses #390 (although it doesn't necessarily fix all instances
of that problem yet).
Github-ref: https://github.com/OfflineIMAP/offlineimap/issues/390
Originally-written-by: Giel van Schijndel <me@mortis.eu>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
If the map file is not properly written (e.g. due to unexpected kill)
offlineimap might wrongly consider some UIDs to have been deleted from the local
side which could lead to data loss.
Use a temporary map file rather than writing to the map file directly.
Github-ref: https://github.com/OfflineIMAP/offlineimap/issues/380
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Do not call resizeterm(), unless is_term_resized() returns True. This
breaks the busy-loop where resizeterm() pushes a KEY_RESIZE onto the
FIFO causing the screen to be redrawn indefinitely (Issue #290).
Also, clear and refresh the main window after it has been resized. This
hopefully fixes the problem where Blinkenlights UI becomes unreadable
after terminal resize (Issue #160).
Closes#160: blinkenlights display is broken
Closes#290: ncurses flicker with blinkenlights UI
Bug-Debian: https://bugs.debian.org/671087
Bug-Debian: https://bugs.debian.org/809676
Signed-off-by: Ilias Tsitsimpis <i.tsitsimpis@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The `threadutil` module depends on the `ui` module, which cycle depends
on `threadutil` through the `Curses` module.
The `ui` module had been loaded before the `threadutil` module, breaking
the above circular dependency, but this was changed in a recent code
refactoring, preventing the `Curses` module from being loaded with the
following exception:
File "./offlineimap.py", line 23, in <module>
from offlineimap import OfflineImap
File "offlineimap/offlineimap/__init__.py", line 20, in <module>
from offlineimap.init import OfflineImap
File "offlineimap/offlineimap/init.py", line 30, in <module>
from offlineimap import globals, threadutil, accounts, folder, mbnames
File "offlineimap/offlineimap/threadutil.py", line 25, in <module>
from offlineimap.ui import getglobalui
File "offlineimap/offlineimap/ui/__init__.py", line 28, in <module>
from offlineimap.ui import Curses
File "offlineimap/offlineimap/ui/Curses.py", line 27, in <module>
from offlineimap.threadutil import ExitNotifyThread
ImportError: cannot import name ExitNotifyThread
Fix the above error by ensuring that the `ui` module gets loaded before
the `threadutil` module. Please note that the above fix is temporary,
and we should refactor the modules to not have circular dependencies.
Bug-Debian: https://bugs.debian.org/839862
Reported-by: Neil McGovern <neilm@debian.org>
Signed-off-by: Ilias Tsitsimpis <i.tsitsimpis@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
BaseFolder now exposes an __unicode__ method so that function needing
unicode transcoding don't crash due to ascii encoding errors.
Signed-off-by: Stéphane Albert <sheeprine@oh.its.fake.nullplace.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Don't pass the account name to the function returned by eval. This allows the
user to define his own arguments.
This fix the code according to the documentation provided in offlineimap.conf.
Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/372
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
To further ensure that messages are synchronized strictly in UID
order, this option can be set to cause only one thread to be used
to synchronise an individual folder, though other folders may
be synchronized simultaneously by other threads.
Signed-off-by: James E. Blair <corvus@gnu.org>
Based-on-patch-by: James E. Blair <corvus@gnu.org>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Return a sorted list of UIDs in getmessageuidlist.
Some MUAs (and their users) like to display messages in UID order
(or "order received"). If offlineimap is used in IMAP<->IMAP mode,
then the order messages are received by the second IMAP server will
be different because offlineimap iterates over a UID list produced
from the keys of a dictionary, which is unsorted.
This change sorts that list of UIDs so that both IMAP servers will
have their messages in the same order (except those times where
messages are appended to folders on both repositories between
syncs).
Signed-off-by: James E. Blair <corvus@gnu.org>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This suppressing the warning generated when running offlineimap on
python2 with -3 switch:
offlineimap/folder/Base.py:29: DeprecationWarning: Overriding __eq__
blocks inheritance of __hash__ in 3.x
Since this object is mutable it should not be hashable.
From Python documentation[1]:
A class that overrides __eq__() and does not define __hash__() will
have its __hash__() implicitly set to None.
Therefore old behaviour is preserved.
[1] https://docs.python.org/3/reference/datamodel.html
Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
It is more readable and returns a list.
This continues work done in 19c4330.
Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This was causing the error message "not all arguments converted during
string formatting." Specifically:
```
% offlineimap -u machineui
OfflineIMAP 7.0.4
Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
msg:protocol:MainThread:7.2.0
msg:initbanner:MainThread:OfflineIMAP+7.0.4%0A++Licensed+under+the+GNU+GPL+v2+or+any+later+version+%28with+an+OpenSSL+exception%29
msg:registerthread:Account+sync+MYHOST:MYHOST
msg:acct:Account+sync+MYHOST:MYHOST
error::Account+sync+MYHOST:ERROR%3A+While+attempting+to+sync+account+%27MYHOST%27%0A++not+all+arguments+converted+during+string+formatting
msg:acctdone:Account+sync+MYHOST:MYHOST
msg:threadExited:MainThread:Account+sync+MYHOST
msg:unregisterthread:MainThread:Account+sync+MYHOST
msg:terminate:MainThread:0%0A%0A
```
Signed-off-by: Christopher League <league@contrapunctus.net>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Otherwise, it might be impossible to know which account is connecting when more
than one is syncing.
Code style.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
They introduce a regression not allowing to discard the XOAUTH2 method when
expected.
The default lambda did not take the "account_name" argument.
Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/362
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
OfflineIMAP frequently delivers mail files to the Maildir consisting exclusively
of a single ASCII digit in IDLE mode. IMAPFolder.getmessage expects 'data' to be
of the form
[(fetch-info, message-body)]
However, the imapobj.uid call in getmessage returns a list of *all* pending
untagged FETCH responses. If any message flags were changed in the selected
IMAP folder since the last command (by another client or another thread in
OfflineIMAP itself), the IMAP server will issue unsolicited FETCH responses
indicating these flag changes (RFC3501, section 7). When this happens, 'data'
will look like, for example
['1231 (FLAGS (\\Seen) UID 5300)',
'1238 (FLAGS (\\Seen) UID 5318)',
('1242 (UID 5325 BODY[] {7976}', message-body)]
Unfortunately, getmessage retrieves the message body as data[0][1], which in
this example is just the string "2", and this is what gets stored in the mail
file.
Multi-threaded OfflineIMAP with IDLE or holdconnectionopen is particularly
susceptible to this problem because flag changes synced back to the IMAP server
on one thread will appear as unsolicited FETCH responses on another thread if it
happens to have the same folder selected. This can also happen without IDLE or
holdconnectionopen or even in single-threaded OfflineIMAP with concurrent access
from other IMAP clients (webmail clients, etc.), though the window for the bug
is much smaller.
Ideally, either imaplib2 or getmessage would parse the fetch responses to find
the response for the requested UID. However, since IMAP only specifies
unilateral FETCH responses for flag changes, it's almost certainly safe to
simply find the element of 'data' that is a tuple (perhaps aborting if there is
more than one tuple) and use that.
Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/162
Based-on-patch-by: Austin Clements <amdragon@MIT.EDU>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Commit 6b28071b0f makes the check once nametrans is applied. This is wrong
because separators are valid characters in names returned by nametrans. E.g:
"Sent" -> "[Gmail]/Sent"
Make the check against the raw name of the folder, instead.
Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/353
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
1. There is one database per folder and sqlite requires to serialize the
writings. Instead of locking at LocalStatusSQLiteFolder object level, introduce
a new DatabaseFileLock object which is shared across threads. This fixes the
concurrent writes issues that some users might experience by duplications or
flags restored to the previous state.
2. Close the database only when we are sure no other threads will use the
connection on a *per-file* basis. Previous fix 677afb8d8f is wrong
because the same lock is shared for all the databases.
Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/350
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
1. There is one database per folder and sqlite requires to serialize the
writings. Instead of locking at LocalStatusSQLiteFolder instance level,
introduce a new DatabaseFileLock object which is shared across threads. This
fixes the concurrent writes issues that some users might experience by
duplications or flags restored to the previous state.
2. Close the database only when we are sure no other threads will use the
connection on a *per-file* basis. Previous fix 677afb8d8f is wrong
because the same lock is shared for all the database files.
Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/350
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This did not work and is not usefull since the purge() method was introduced in
1410a391bc. Actually, the purge() does what deletemessagelist() was supposed to
achieve.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
It's required to have more than one connection to the database for the
maxconnections configuration option to work with threads. However,
connection.close() is closing all the connections. Only close the connection
when no more thread need it.
Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/350
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
It's required to have more than one connection to the database for the
maxconnections configuration option to work with threads. However,
connection.close() is closing all the connections. Only close the connection
when no more thread need it.
Backported-from: 856b74407bd7f634cae5a8c2d9b84e13d14c12d2
Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/350
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This is usefull to remove dangling entries for removed accounts or if mbnames is
not enabled anymore.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Output a warning so that we can dig into this. For now, the patch doesn't fix
the root cause. If the server returns UID 0 as valid UID number, this must be
ignored as soon as possible.
Github-ref: https://github.com/OfflineIMAP/offlineimap/issues/336
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
When a maildir is removed it must be considered new for the sync. However, the
local cache of the folder remains. This means the sync of the folder removes all
the missing emails.
Avoid loosing of data for users not aware of the local cache by removing any
pre-existing status cache of a folder when we actually want to create the
database.
Improve style.
Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/333
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
- Learn to support UTF-8 characters where it was not supported for usernames and
passwords (but for netrc).
- Fix the types in the code for both py2 and py3: we now expect unicode for
usernames and passwords.
Unicode (UTF-8) is required only for variables with non-ASCII characters.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Make mbnames to work with intermediate files, one per account, in the JSON
format. The mbnames target is built from those intermediate files.
Github-Fix: https://github.com/OfflineIMAP/offlineimap/issues/66
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
- Expose more literals.
- Fix a type to bytes.
- Fix comment.
Changes submitted upstream by Łukasz and me.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Backported from 8e995a69bfa003ab822b55731429d84b3bc5626f.
We currently close the database as soon as possible while we handle the status
backend but this is still too early because autorefresh induces looping on this
code block.
Instead of delaying the closing outside of the loop, it's easier to delay the
opening as late as possible (inside the loop). The downside is that the database
is opened/closed more than once when autorefresh is enabled. The good news is
that this make the code much easier.
Fixes regression introduces by 6fb5700.
Reported-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The virtual imaplib2 first try to import imaplib2 when provided by the system.
If not provided or if the version is not supported, fallback on the bundled
imaplib2 version.
Distribution maintainers can now easily remove the bundled imaplib2 version if
they want to get it packaged outside of offlineimap.
We still want to provide imaplib2 by default because:
- this library is neither in Python core nor packaged by a lot of distributions;
- users expect to be able to run offlineimap by just downloading the tarball or
after a git clone.
In order to avoid unexpected (too old) versions of imaplib2, we restrict the
supported versions of this librabry.
Reviewed-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
