Commit Graph

148 Commits

Author SHA1 Message Date
Rodolfo García Peñas (kix)
df28136bb3 accounts.py Removed extra spaces after equal
I changed the intentation to join the equal with the quote.
2020-08-30 18:19:05 +02:00
Rodolfo García Peñas (kix)
1a529b6b0e offlineimap/ files singleton-comparison
This patch change these errors in the offlineimap folder

C0121: Comparison to None should be 'expr is None' (singleton-comparison)
C0121: Comparison to None should be 'expr is not None' (singleton-comparison)
2020-08-30 11:27:03 +02:00
Rodolfo García Peñas (kix)
9f15c05520 Reformat offlineimap/accounts.py
Add some spaces, remove lines,... now format is better (lintian).
2020-08-29 20:05:18 +02:00
Nicolas Sebrecht
801893cc73 make portalocker really optional
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2019-04-27 14:56:17 +02:00
kimim
2806f40071 use portable locker to support cygwin in Windows
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2019-04-27 14:37:44 +02:00
Nicolas Sebrecht
a51064e83a maxage: always compute the remote cache list for min_uid
Github-ref: https://github.com/OfflineIMAP/offlineimap/issues/581
Tested-by: https://github.com/ianbrody
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2018-12-11 09:29:37 +01:00
Nicolas Sebrecht
0d6a9a44da maxage: don't consider negative UIDs when computing min UID
With new emails we could have negative UIDs in come use cases. Exclude these
from the list of UIDs. The negative UIDs lead to invalid SEARCH command:

 SEARCH command error: BAD ['Could not parse command']. Data: FMAO19 SEARCH (UID -4:*)

Github-ref: https://github.com/OfflineIMAP/offlineimap/issues/512
Tested-by: https://github.com/shubhamkrm
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2017-12-22 22:01:00 +01:00
Nicolas Sebrecht
6b64f87462 provide non-zero exit code & display configuration mismatch error at the end of the run
This behaviour is what users would expected most. The error message is about
utf8foldernames and decodefoldernames.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2017-10-02 21:24:12 +02:00
Nicolas Sebrecht
12de158ca2 Merge branch 'uliska/utf8foldernames' into next 2017-10-02 21:11:29 +02:00
Urs Liska
8b398f3aa8 utf8: Safeguard against use of old *and* new option
The new 'utf8foldernames' will not work together with the existing
'decodefoldernames' option (which will be documented in the next
commit). Therefore this commit will check for this condition and
abort the synchronization of a misconfigured account before doing
any changes.
Other accounts are not affected.

Signed-off-by: Urs Liska <git@ursliska.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2017-10-02 21:09:43 +02:00
Urs Liska
36d726763d utf8: implement utf8foldernames option
If utf8foldernames is enabled on account level all folder names read
from the IMAP server will immediately be reencoded to UTF-8. Names
will be treated as UTF-8 as long as the IMAP server isn't contacted again,
for which they are reencoded to IMAP4-UTF-7.

This means that any further processing such as nametrans, folderfilter
etc. will act upon the UTF-8 names, which will have to be documented
carefully.

NOTE 1:
GMail repositories and folders inherit from the IMAP... classes, so I don't
know yet if these changes have ugly side-effects. But web research suggests
that GMail IMAP folders are equally encoded in UTF-7 so that should work
identically here and incorporate the same improvements.

NOTE 2:
I could not test the behaviour with idlefolders as I didn't get this option
to work at all, not even with the latest stable version.

NOTE 3:
I *did* test to sync an IMAP repository against another IMAP repository.

Signed-off-by: Urs Liska <git@ursliska.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2017-10-02 21:09:43 +02:00
Nicolas Sebrecht
31eee55672 accounts: error out when no folder to sync
Such error can be hit when the folderfilter is wrong and excludes all the
folders.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2017-10-02 02:43:47 +02:00
Nicolas Sebrecht
f37b97c61b maxage: use the remote folder first to compute min_uid
On each folder scan, we must compute the min_uid. Locally, this is done by
relying on the prefix timestamp in the filenames. If for whatever reason they do
not reflect the Date header of the email, changing maxage leads to undefined
behaviour.

To prevent from this, we rather rely on the remote folder. We assume that the
remotes are more reliable to provide correct sets of UIDs based on dates than we
are.

Github-ref: https://github.com/OfflineIMAP/offlineimap/issues/384
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2017-05-21 00:50:04 +02:00
Nicolas Sebrecht
5d5ad62fa7 mbnames: don't duplicate entries in autorefresh mode
mbnames is initialized and written once in the run from OfflineImap.__sync().

However, the in-memory instance is fed with data at sync time for each folder
and the intermediate files are written as soon as all the folders are synced for
the account. All of this is done inside the SyncableAccount.__sync() method
while the syncrunner is looping on this.

This means that we duplicate entries for mbnames in each loop (most likely when
autorefresh is enabled).

It is wrong to have duplicates in mbnames for each account. Ignore duplicates
when adding data in the mbnames intermediate files.

Github-ref: https://github.com/OfflineIMAP/offlineimap/issues/467
Reported-and-tested-by: Shin Kojima <shin@kojima.org>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2017-05-19 08:32:22 +02:00
Nicolas Sebrecht
fc079499e9 prevent synchronization of identical folders from multiple threads
Use a mutex to lock the sync transaction, uniquely identified by the remote
folder name.

Serializing the syncs prevents any IDLE thread from syncing the same folder of
a concurrent autorefresh thread. Othewise, we could download new messages
twice, or read "one message past the last one" which in some cases creates a
duplicated messages.  Also prevents IDLE threads from reading new messages
before the autorefresh thread finishes synchronization.

Github-ref: #421
Originally-submitted-by: Michael Hohmuth <hohmuth@sax.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2017-02-28 19:16:18 +01:00
Nicolas Sebrecht
54bbe79930 accounts: minor code refactoring
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-11-20 20:22:37 +01:00
Abdo Roig-Maranges
c4fcd9f7f2 fix remaining instance of check_uid_validity refactoring
Signed-off-by: Abdo Roig-Maranges <abdo.roig@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-11-10 11:24:33 +01:00
Nicolas Sebrecht
5eb131ef2b accounts: minor refactoring
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-11-08 07:11:30 +01:00
Nicolas Sebrecht
e452c344d9 minor code refactoring
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-11-07 23:03:40 +01:00
Nicolas Sebrecht
9606cfcfc1 don't allow negative values for autorefresh
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-11-07 23:01:15 +01:00
Nicolas Sebrecht
203c28f21f add the repository name when connecting
Otherwise, it might be impossible to know which account is connecting when more
than one is syncing.

Code style.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-07-30 01:44:27 +02:00
Nicolas Sebrecht
7fe90b8e0d properly ignore folders with invalid characters (sep) in their name
Commit 6b28071b0f makes the check once nametrans is applied. This is wrong
because separators are valid characters in names returned by nametrans. E.g:

  "Sent" -> "[Gmail]/Sent"

Make the check against the raw name of the folder, instead.

Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/353
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-07-27 18:45:18 +02:00
Nicolas Sebrecht
3c42913120 sqlite: properly serialize operations on the databases
1. There is one database per folder and sqlite requires to serialize the
writings. Instead of locking at LocalStatusSQLiteFolder instance level,
introduce a new DatabaseFileLock object which is shared across threads. This
fixes the concurrent writes issues that some users might experience by
duplications or flags restored to the previous state.

2. Close the database only when we are sure no other threads will use the
connection on a *per-file* basis. Previous fix 677afb8d8f is wrong
because the same lock is shared for all the database files.

Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/350
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-07-26 01:57:25 +02:00
Nicolas Sebrecht
5aa2a883f0 local status: deletemessagelist: remove broken code
This did not work and is not usefull since the purge() method was introduced in
1410a391bc. Actually, the purge() does what deletemessagelist() was supposed to
achieve.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-07-25 15:06:39 +02:00
Nicolas Sebrecht
c992f58530 offlineimap.conf: allow non-spaces in the account list
Regression introduced in v7.0.0.

Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/355
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-07-24 19:39:43 +02:00
Nicolas Sebrecht
2e81331a3c learn --delete-folder CLI option
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-07-08 21:34:25 +02:00
Nicolas Sebrecht
bb8e03bced init: factorize code to get active accounts
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-07-08 21:34:25 +02:00
Nicolas Sebrecht
7945e10a76 correctly reraise errors with six
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-07-03 19:56:14 +02:00
Nicolas Sebrecht
6b28071b0f don't sync folders with local separator characters in their names
Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/148
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-07-03 19:52:34 +02:00
Nicolas Sebrecht
1410a391bc avoid removing of data when user removed a maildir
When a maildir is removed it must be considered new for the sync. However, the
local cache of the folder remains. This means the sync of the folder removes all
the missing emails.

Avoid loosing of data for users not aware of the local cache by removing any
pre-existing status cache of a folder when we actually want to create the
database.

Improve style.

Github-fix: https://github.com/OfflineIMAP/offlineimap/issues/333
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-06-26 17:00:23 +02:00
Nicolas Sebrecht
092264c8e7 fix: when called with -a, mbnames must not erase entries of other accounts
Make mbnames to work with intermediate files, one per account, in the JSON
format. The mbnames target is built from those intermediate files.

Github-Fix: https://github.com/OfflineIMAP/offlineimap/issues/66
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-06-25 17:33:42 +02:00
Nicolas Sebrecht
e3fe848c6b minor code cleanups
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-06-08 19:03:02 +02:00
Nicolas Sebrecht
e0fdcb3852 threading: improve variable names and factorize code
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-05-18 04:23:02 +02:00
Łukasz Żarnowiecki
0addcbabf0 py3: raise exceptions using six module
There is no other way to make Python2 and Python3 happy, because syntax
raise E, V, T is incompatible with the latter.

Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-05-18 01:49:09 +02:00
Nicolas Sebrecht
83539f8601 threading: minor improvements
Factorize string, enhance comments and minor code improvements.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-05-18 01:32:06 +02:00
Nicolas Sebrecht
05c75e8c8d sqlite: open database when we use it rather than at instantiation time
We currently close the database as soon as possible while we handle the status
backend but this is still too early because autorefresh induces looping on this
code block.

Instead of delaying the closing outside of the loop, it's easier to delay the
opening as late as possible (inside the loop). The downside is that the database
is opened/closed more than once when autorefresh is enabled. The good news is
that this make the code much easier.

Fixes regression introduces by 6fb5700.

Reported-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-05-12 18:51:59 +02:00
Nicolas Sebrecht
2611a0ba6a threading: add comments
Simplify the code.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-05-12 04:35:21 +02:00
Łukasz Żarnowiecki
19c43308b9 Change filter with lambda to list comprehension
It is more readable and returns a list therefore it is compatible both
with Python 2 and 3.

Signed-off-by: Łukasz Żarnowiecki <dolohow@outlook.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-05-10 02:53:09 +02:00
Nicolas Sebrecht
6fb5700f94 SQLite: close db when done
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-04-09 19:55:26 +02:00
Nicolas Sebrecht
1ba5e9160d minor: logs: print readonly message in all debug modes
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-04-09 18:57:00 +02:00
Nicolas Sebrecht
801dda521e accounts.py: minor improvements
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2016-04-09 18:50:12 +02:00
Nicolas Sebrecht
c3f3012cfc accounts: remove unused imaplibutil import
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-04-07 16:34:35 +02:00
Janna Martl
8096f6cd5b maxage: fix timezone issues, remove IMAP-IMAP support, add startdate option
1. When using maxage, local and remote messagelists are supposed to only
contain messages from at most maxage days ago. But local and remote used
different timezones to calculate what "maxage days ago" means, resulting
in removals on one side. Now, we ask the local folder for maxage days'
worth of mail, find the lowest UID, and then ask the remote folder for
all UID's starting with that lowest one.

2. maxage was fundamentally wrong in the IMAP-IMAP case: it assumed that
remote messages have UIDs in the same order as their local counterparts,
which could be false, e.g. when messages are copied in quick succession.
So, remove support for maxage in the IMAP-IMAP case.

3. Add startdate option for IMAP-IMAP syncs: use messages from the given
repository starting at startdate, and all messages from the other
repository. In the first sync, the other repository must be empty.

4. Allow maxage to be specified either as number of days to sync (as
previously) or as a fixed date.

Signed-off-by: Janna Martl <janna.martl109@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-04-07 11:54:35 +02:00
Ben Boeckel
4217fccb82 mbnames: add option to write the file once per account
The basic problem is in the context of syncing multiple accounts where
one is fast and the others are slower (due to the number of folders).
When the fast account completes, the other accounts are partially written
through the list and if the file is read during this time, the list can
be useless. However, in the general case, the file is probably left
around from a previous run of offlineimap and is more correct, so add an
option to leave it alone until all syncing is done.

Incremental is still the default since this running offlineimap using
its own timer setup is likely the most common setup. Turning it off
works best with one-shot mode triggered by cron or systemd timers.

Signed-off-by: Ben Boeckel <mathstuf@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-03-22 21:15:46 +01:00
Nicolas Sebrecht
461554b7b1 more style consistency
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-02-10 17:25:00 +01:00
Nicolas Sebrecht
41fa3ae4fc more style consistency
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-13 18:16:25 +01:00
Giovanni Mascellani
29e9b7ab39 Drop caches after having processed folders.
This enhances a lot memory consumption when you have many folders
to process.

Signed-off-by: Giovanni Mascellani <mascellani@poisson.phc.unipi.it>
2015-01-12 17:17:42 +01:00
Nicolas Sebrecht
63e499c6f2 more consistent style
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-12 17:17:42 +01:00
Nicolas Sebrecht
a44718130d minor: add comments
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-12 17:17:42 +01:00
Eygene Ryabinkin
e7fabf9e57 Properly re-raise exception to save original tracebacks
We usually mutate some exceptions to OfflineImapError() and it is
a whole lot better if such exception will show up with the original
traceback, so all valid occurrences of such mutations were transformed
to the 3-tuple form of "raise".  Had also added coding guidelines
document where this re-raise strategy is documented.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2015-01-12 17:51:21 +03:00