tobias/docker-offlineimap
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branch 'next'

Browse Source
This commit is contained in:
Sebastian Spaeth 2012-06-02 13:41:46 +02:00
commit 010941e12e
14 changed files with 134 additions and 217 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
Changelog.rst
View File

@ -8,14 +8,15 @@ ChangeLog
WIP (add new stuff for the next release)
========================================
New Features
------------
OfflineIMAP v6.5.4 (2012-06-02)
=================================
Changes
-------
Bug Fixes
---------
* bump bundled imaplib2 library 2.29 --> 2.33
* Actually perform the SSL fingerprint check (reported by J. Cook)
* Curses UI, don't use colors after we shut down curses already (C.Höger)
* Document that '%' needs encoding as '%%' in *.conf
* Fix crash when IMAP.quickchanged() led to an Error (reported by sharat87)
* Implement the createfolders setting to disable folder propagation (see docs)
OfflineIMAP v6.5.3.1 (2012-04-03)
=================================

182
docs/dev-doc-src/nametrans.rst
View File

@ -1,182 +0,0 @@
.. _folder_filtering_and_name_translation:
Folder filtering and Name translation
=====================================
OfflineImap provides advanced and potentially complex possibilities for
filtering and translating folder names. If you don't need any of this, you can
safely skip this section.
.. warning::
Starting with v6.4.0, OfflineImap supports the creation of folders on the remote repostory. This change means that people that only had a nametrans option on the remote repository (everyone) will need to have a nametrans setting on the local repository too that will reverse the name transformation. See section `Reverse nametrans`_ for details.
folderfilter
------------
If you do not want to synchronize all your filters, you can specify a `folderfilter`_ function that determines which folders to include in a sync and which to exclude. Typically, you would set a folderfilter option on the remote repository only, and it would be a lambda or any other python function.
The only parameter to that function is the folder name. If the filter
function returns True, the folder will be synced, if it returns False,
it. will be skipped. The folderfilter operates on the *UNTRANSLATED*
name (before any `nametrans`_ fudging takes place). Consider the
examples below to get an idea of what they do.
Example 1: synchronizing only INBOX and Sent::
folderfilter = lambda folder: folder in ['INBOX', 'Sent']
Example 2: synchronizing everything except Trash::
folderfilter = lambda folder: folder not in ['Trash']
Example 3: Using a regular expression to exclude Trash and all folders
containing the characters "Del"::
folderfilter = lambda folder: not re.search('(^Trash$|Del)', folder)
.. note::
If folderfilter is not specified, ALL remote folders will be
synchronized.
You can span multiple lines by indenting the others. (Use backslashes
at the end when required by Python syntax) For instance::
folderfilter = lambda foldername: foldername in
['INBOX', 'Sent Mail', 'Deleted Items',
'Received']
Usually it suffices to put a `folderfilter`_ setting in the remote repository section. You might want to put a folderfilter option on the local repository if you want to prevent some folders on the local repository to be created on the remote one. (Even in this case, folder filters on the remote repository will prevent that)
folderincludes
^^^^^^^^^^^^^^
You can specify `folderincludes`_ to manually include additional folders to be synced, even if they had been filtered out by a folderfilter setting. `folderincludes`_ should return a Python list.
This can be used to 1) add a folder that was excluded by your
folderfilter rule, 2) to include a folder that your server does not specify
with its LIST option, or 3) to include a folder that is outside your basic
`reference`. The `reference` value will not be prefixed to this folder
name, even if you have specified one. For example::
folderincludes = ['debian.user', 'debian.personal']
This will add the "debian.user" and "debian.personal" folders even if you
have filtered out everything starting with "debian" in your folderfilter
settings.
nametrans
----------
Sometimes, folders need to have different names on the remote and the
local repositories. To achieve this you can specify a folder name
translator. This must be a eval-able Python expression that takes a
foldername arg and returns the new value. We suggest a lambda function,
but it could be any python function really. If you use nametrans rules, you will need to set them both on the remote and the local repository, see `Reverse nametrans`_ just below for details. The following examples are thought to be put in the remote repository section.
The below will remove "INBOX." from the leading edge of folders (great
for Courier IMAP users)::
nametrans = lambda folder: re.sub('^INBOX\.', '', folder)
Using Courier remotely and want to duplicate its mailbox naming
locally? Try this::
nametrans = lambda folder: re.sub('^INBOX\.*', '.', folder)
.. warning::
You MUST construct nametrans rules such that it NEVER returns the
same value for two folders, UNLESS the second values are filtered
out by folderfilter below. That is, two filters on one side may
never point to the same folder on the other side. Failure to follow
this rule will result in undefined behavior. See also *Sharing a
maildir with multiple IMAP servers* in the :ref:`pitfalls` section.
Reverse nametrans
^^^^^^^^^^^^^^^^^^
Since 6.4.0, OfflineImap supports the creation of folders on the remote repository and that complicates things. Previously, only one nametrans setting on the remote repository was needed and that transformed a remote to a local name. However, nametrans transformations are one-way, and OfflineImap has no way using those rules on the remote repository to back local names to remote names.
Take a remote nametrans rule `lambda f: re.sub('^INBOX/','',f)` which cuts of any existing INBOX prefix. Now, if we parse a list of local folders, finding e.g. a folder "Sent", is it supposed to map to "INBOX/Sent" or to "Sent"? We have no way of knowing. This is why **every nametrans setting on a remote repository requires an equivalent nametrans rule on the local repository that reverses the transformation**.
Take the above examples. If your remote nametrans setting was::
nametrans = lambda folder: re.sub('^INBOX\.', '', folder)
then you will want to have this in your local repository, prepending "INBOX" to any local folder name::
nametrans = lambda folder: 'INBOX' + folder
Failure to set the local nametrans rule will lead to weird-looking error messages of -for instance- this type::
ERROR: Creating folder moo.foo on repository remote
Folder 'moo.foo'[remote] could not be created. Server responded: ('NO', ['Unknown namespace.'])
(This indicates that you attempted to create a folder "Sent" when all remote folders needed to be under the prefix of "INBOX.").
OfflineImap will make some sanity checks if it needs to create a new
folder on the remote side and a back-and-forth nametrans-lation does not
yield the original foldername (as that could potentially lead to
infinite folder creation cycles).
You can probably already see now that creating nametrans rules can be a pretty daunting and complex endeavour. Check out the Use cases in the manual. If you have some interesting use cases that we can present as examples here, please let us know.
Debugging folderfilter and nametrans
------------------------------------
Given the complexity of the functions and regexes involved, it is easy to misconfigure things. One way to test your configuration without danger to corrupt anything or to create unwanted folders is to invoke offlineimap with the `--info` option.
It will output a list of folders and their transformations on the screen (save them to a file with -l info.log), and will help you to tweak your rules as well as to understand your configuration. It also provides good output for bug reporting.
FAQ on nametrans
----------------
Where to put nametrans rules, on the remote and/or local repository?
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
If you never intend to create new folders on the LOCAL repository that
need to be synced to the REMOTE repository, it is sufficient to create a
nametrans rule on the remote Repository section. This will be used to
determine the names of new folder names on the LOCAL repository, and to
match existing folders that correspond.
*IF* you create folders on the local repository, that are supposed to be
automatically created on the remote repository, you will need to create
a nametrans rule that provides the reverse name translation.
(A nametrans rule provides only a one-way translation of names and in
order to know which names folders on the LOCAL side would have on the
REMOTE side, you need to specify the reverse nametrans rule on the local
repository)
OfflineImap will complain if it needs to create a new folder on the
remote side and a back-and-forth nametrans-lation does not yield the
original foldername (as that could potentially lead to infinite folder
creation cycles).
What folder separators do I need to use in nametrans rules?
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
**Q:** If I sync from an IMAP server with folder separator '/' to a
Maildir using the default folder separator '.' which do I need to use
in nametrans rules?::
nametrans = lambda f: "INBOX/" + f
or::
nametrans = lambda f: "INBOX." + f
**A:** Generally use the folder separator as defined in the repository
you write the nametrans rule for. That is, use '/' in the above
case. We will pass in the untranslated name of the IMAP folder as
parameter (here `f`). The translated name will ultimately have all
folder separators be replaced with the destination repositories'
folder separator.
So if 'f' was "Sent", the first nametrans yields the translated name
"INBOX/Sent" to be used on the other side. As that repository uses the
folder separator '.' rather than '/', the ultimate name to be used will
be "INBOX.Sent".
(As a final note, the smart will see that both variants of the above
nametrans rule would have worked identically in this case)

16
docs/doc-src/advanced_config.rst Normal file
View File

@ -0,0 +1,16 @@
Message filtering
=================
There are two ways to selectively filter messages out of a folder, using `maxsize` and `maxage`. Setting each option will basically ignore all messages that are on the server by pretending they don't exist.
:todo: explain them and give tipps on how to use and not use them. Use cases!
maxage
------
:todo: !
maxsize
-------
:todo: !

4
docs/doc-src/index.rst
View File

@ -17,8 +17,11 @@ More information on specific topics can be found on the following pages:
**User documentation**
* :doc:`Overview and features <features>`
* :doc:`installation/uninstall <INSTALL>`
**Configuration**
* :doc:`user manual/Configuration <MANUAL>`
* :doc:`Folder filtering & name transformation guide <nametrans>`
* :doc:`maxage <advanced_config>`
* :doc:`command line options <offlineimap>`
* :doc:`Frequently Asked Questions <FAQ>`
@ -34,6 +37,7 @@ More information on specific topics can be found on the following pages:
INSTALL
MANUAL
nametrans
advanced_config
offlineimap
FAQ

16
docs/doc-src/nametrans.rst
View File

@ -65,6 +65,22 @@ have filtered out everything starting with "debian" in your folderfilter
settings.
createfolders
-------------
By default OfflineImap propagates new folders in both
directions. Sometimes this is not what you want. E.g. you might want
new folders on your IMAP server to propagate to your local MailDir,
but not the other way around. The 'readonly' setting on a repository
will not help here, as it prevents any change from occuring on that
repository. This is what the `createfolders` setting is for. By
default it is `True`, meaning that new folders can be created on this
repository. To prevent folders from ever being created on a
repository, set this to `False`. If you set this to False on the
REMOTE repository, you will not have to create the `Reverse
nametrans`_ rules on the LOCAL repository.
nametrans
----------

11
offlineimap.conf
View File

@ -21,6 +21,7 @@
#
# would set the trashfolder setting for your German gmail accounts.
# NOTE2: This implies that any '%' needs to be encoded as '%%'
##################################################
# General definitions
@ -341,7 +342,7 @@ remoteuser = username
# OfflineIMAP starts when using a UI that supports this.
#
# 2. The remote password stored in this file with the remotepass
# option. Example:
# option. Any '%' needs to be encoded as '%%'. Example:
# remotepass = mypassword
#
# 3. The remote password stored as a single line in an external
@ -503,6 +504,14 @@ remoteuser = username
# one. For example:
# folderincludes = ['debian.user', 'debian.personal']
# If you do not want to have any folders created on this repository,
# set the createfolders variable to False, the default is True. Using
# this feature you can e.g. disable the propagation of new folders to
# the new repository.
#createfolders = True
# You can specify 'foldersort' to determine how folders are sorted.
# This affects order of synchronization and mbnames. The expression
# should return -1, 0, or 1, as the default Python cmp() does. The two

2
offlineimap/__init__.py
View File

@ -1,7 +1,7 @@
__all__ = ['OfflineImap']
__productname__ = 'OfflineIMAP'
__version__ = "6.5.3.1"
__version__ = "6.5.4"
__copyright__ = "Copyright 2002-2012 John Goerzen & contributors"
__author__ = "John Goerzen"
__author_email__= "john@complete.org"

7
offlineimap/folder/IMAP.py
View File

@ -93,14 +93,17 @@ class IMAPFolder(BaseFolder):
# Select folder and get number of messages
restype, imapdata = imapobj.select(self.getfullname(), True,
True)
self.imapserver.releaseconnection(imapobj)
except OfflineImapError as e:
# retry on dropped connections, raise otherwise
self.imapserver.releaseconnection(imapobj, True)
if e.severity == OfflineImapError.ERROR.FOLDER_RETRY:
retry = True
else: raise
finally:
self.imapserver.releaseconnection(imapobj)
except:
# cleanup and raise on all other errors
self.imapserver.releaseconnection(imapobj, True)
raise
# 1. Some mail servers do not return an EXISTS response
# if the folder is empty. 2. ZIMBRA servers can return
# multiple EXISTS replies in the form 500, 1000, 1500,

39
offlineimap/imaplib2.py
View File

@ -17,9 +17,9 @@ Public functions: Internaldate2Time
__all__ = ("IMAP4", "IMAP4_SSL", "IMAP4_stream",
"Internaldate2Time", "ParseFlags", "Time2Internaldate")
__version__ = "2.29"
__version__ = "2.33"
__release__ = "2"
__revision__ = "29"
__revision__ = "33"
__credits__ = """
Authentication code contributed by Donn Cave <donn@u.washington.edu> June 1998.
String method conversion by ESR, February 2001.
@ -462,19 +462,16 @@ class IMAP4(object):
cert_reqs = ssl.CERT_NONE
self.sock = ssl.wrap_socket(self.sock, self.keyfile, self.certfile, ca_certs=self.ca_certs, cert_reqs=cert_reqs)
ssl_exc = ssl.SSLError
self.read_fd = self.sock.fileno()
except ImportError:
# No ssl module, and socket.ssl does not allow certificate verification
if self.ca_certs is not None:
raise socket.sslerror("SSL CA certificates cannot be checked without ssl module")
self.sock = socket.ssl(self.sock, self.keyfile, self.certfile)
ssl_exc = socket.sslerror
# No ssl module, and socket.ssl has no fileno(), and does not allow certificate verification
raise socket.sslerror("imaplib2 SSL mode does not work without ssl module")
if self.cert_verify_cb is not None:
cert_err = self.cert_verify_cb(self.sock.getpeercert(), self.host)
if cert_err:
raise ssl_exc(cert_err)
self.read_fd = self.sock.fileno()
def start_compressing(self):
@ -496,7 +493,7 @@ class IMAP4(object):
if self.decompressor.unconsumed_tail:
data = self.decompressor.unconsumed_tail
else:
data = self.sock.recv(8192)
data = self.sock.recv(READ_SIZE)
return self.decompressor.decompress(data, size)
@ -1233,9 +1230,10 @@ class IMAP4(object):
def _choose_nonull_or_dflt(self, dflt, *args):
dflttyp = type(dflt)
if isinstance(dflttyp, basestring):
if isinstance(dflt, basestring):
dflttyp = basestring # Allow any string type
else:
dflttyp = type(dflt)
for arg in args:
if arg is not None:
if isinstance(arg, dflttyp):
@ -1591,7 +1589,8 @@ class IMAP4(object):
def _simple_command(self, name, *args, **kw):
if 'callback' in kw:
self._command(name, *args, callback=self._command_completer, cb_arg=kw, cb_self=True)
# Note: old calling sequence for back-compat with python <2.6
self._command(name, callback=self._command_completer, cb_arg=kw, cb_self=True, *args)
return (None, None)
return self._command_complete(self._command(name, *args), kw)
@ -1752,8 +1751,9 @@ class IMAP4(object):
if rxzero > 5:
raise IOError("Too many read 0")
time.sleep(0.1)
else:
rxzero = 0
continue # Try again
rxzero = 0
while True:
stop = data.find('\n', start)
if stop < 0:
@ -1818,8 +1818,9 @@ class IMAP4(object):
if rxzero > 5:
raise IOError("Too many read 0")
time.sleep(0.1)
else:
rxzero = 0
continue # Try again
rxzero = 0
while True:
stop = data.find('\n', start)
if stop < 0:
@ -2020,7 +2021,7 @@ class IMAP4_SSL(IMAP4):
if self.decompressor.unconsumed_tail:
data = self.decompressor.unconsumed_tail
else:
data = self.sock.read(8192)
data = self.sock.read(READ_SIZE)
return self.decompressor.decompress(data, size)
@ -2047,7 +2048,7 @@ class IMAP4_SSL(IMAP4):
def ssl(self):
"""ssl = ssl()
Return socket.ssl instance used to communicate with the IMAP4 server."""
Return ssl instance used to communicate with the IMAP4 server."""
return self.sock
@ -2103,7 +2104,7 @@ class IMAP4_stream(IMAP4):
if self.decompressor.unconsumed_tail:
data = self.decompressor.unconsumed_tail
else:
data = os.read(self.read_fd, 8192)
data = os.read(self.read_fd, READ_SIZE)
return self.decompressor.decompress(data, size)

5
offlineimap/imaplibutil.py
View File

@ -1,6 +1,6 @@
# imaplib utilities
# Copyright (C) 2002-2007 John Goerzen <jgoerzen@complete.org>
# 2010 Sebastian Spaeth <Sebastian@SSpaeth.de>
# 2012-2012 Sebastian Spaeth <Sebastian@SSpaeth.de>
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
@ -143,8 +143,7 @@ class WrappedIMAP4_SSL(UsefulIMAPMixIn, IMAP4_SSL):
def open(self, host=None, port=None):
super(WrappedIMAP4_SSL, self).open(host, port)
if (self._fingerprint or not self.ca_certs) and\
'ssl' in locals(): # <--disable for python 2.5
if (self._fingerprint or not self.ca_certs):
# compare fingerprints
fingerprint = sha1(self.sock.getpeercert(True)).hexdigest()
if fingerprint != self._fingerprint:

21
offlineimap/repository/Base.py
View File

@ -32,6 +32,7 @@ class BaseRepository(CustomConfig.ConfigHelperMixin, object):
self.name = reposname
self.localeval = account.getlocaleval()
self._accountname = self.account.getname()
self._readonly = self.getconfboolean('readonly', False)
self.uiddir = os.path.join(self.config.getmetadatadir(), 'Repository-' + self.name)
if not os.path.exists(self.uiddir):
os.mkdir(self.uiddir, 0o700)
@ -108,6 +109,11 @@ class BaseRepository(CustomConfig.ConfigHelperMixin, object):
def getconfig(self):
return self.config
@property
def readonly(self):
"""Is the repository readonly?"""
return self._readonly
def getlocaleval(self):
return self.account.getlocaleval()
@ -123,6 +129,13 @@ class BaseRepository(CustomConfig.ConfigHelperMixin, object):
def getsep(self):
raise NotImplementedError
def get_create_folders(self):
"""Is folder creation enabled on this repository?
It is disabled by either setting the whole repository
'readonly' or by using the 'createfolders' setting."""
return self._readonly or self.getconfboolean('createfolders', True)
def makefolder(self, foldername):
"""Create a new folder"""
raise NotImplementedError
@ -141,6 +154,10 @@ class BaseRepository(CustomConfig.ConfigHelperMixin, object):
that forward and backward nametrans actually match up!
Configuring nametrans on BOTH repositories therefore could lead
to infinite folder creation cycles."""
if not self.get_create_folders() and not dst_repo.get_create_folders():
# quick exit if no folder creation is enabled on either side.
return
src_repo = self
src_folders = src_repo.getfolders()
dst_folders = dst_repo.getfolders()
@ -160,7 +177,7 @@ class BaseRepository(CustomConfig.ConfigHelperMixin, object):
# Find new folders on src_repo.
for src_name_t, src_folder in src_hash.iteritems():
# Don't create on dst_repo, if it is readonly
if dst_repo.getconfboolean('readonly', False):
if not dst_repo.get_create_folders():
break
if src_folder.sync_this and not src_name_t in dst_folders:
try:
@ -175,7 +192,7 @@ class BaseRepository(CustomConfig.ConfigHelperMixin, object):
status_repo.getsep()))
# Find new folders on dst_repo.
for dst_name_t, dst_folder in dst_hash.iteritems():
if self.getconfboolean('readonly', False):
if not src_repo.get_create_folders():
# Don't create missing folder on readonly repo.
break

2
offlineimap/ui/Curses.py
View File

@ -613,6 +613,8 @@ class Blinkenlights(UIBase, CursesUtil):
# basic one, so exceptions and stuff are properly displayed
self.logger.removeHandler(self._log_con_handler)
UIBase.setup_consolehandler(self)
# reset the warning method, we do not have curses anymore
self.warn = super(Blinkenlights, self).warn
# finally call parent terminate which prints out exceptions etc
super(Blinkenlights, self).terminate(*args, **kwargs)

3
test/tests/test_00_imaputil.py
View File

@ -59,6 +59,9 @@ class TestInternalFunctions(unittest.TestCase):
res = imaputil.imapsplit(b'(\\HasNoChildren) "." "INBOX.Sent"')
self.assertEqual(res, [b'(\\HasNoChildren)', b'"."', b'"INBOX.Sent"'])
res = imaputil.imapsplit(b'"mo\\" o" sdfsdf')
self.assertEqual(res, [b'"mo\\" o"', b'sdfsdf'])
def test_02_flagsplit(self):
"""Test imaputil.flagsplit()"""
res = imaputil.flagsplit(b'(\\Draft \\Deleted)')

28
test/tests/test_01_basic.py
View File

@ -105,6 +105,7 @@ class TestBasicFunctions(unittest.TestCase):
# Write out default config file again
OLITestLib.write_config_file()
def test_04_createmail(self):
"""Create mail in OLItest 1, sync, wipe folder sync
@ -127,3 +128,30 @@ class TestBasicFunctions(unittest.TestCase):
self.assertFalse (None in uids, msg = "All mails should have been "+ \
"assigned the IMAP's UID number, but {} messages had no valid ID "\
.format(len([None for x in uids if x==None])))
def test_05_createfolders(self):
"""Test if createfolders works as expected
Create a local Maildir, then sync with remote "createfolders"
disabled. Delete local Maildir and sync. We should have no new
local maildir then. TODO: Rewrite this test to directly test
and count the remote folders when the helper functions have
been written"""
config = OLITestLib.get_default_config()
config.set('Repository IMAP', 'createfolders',
'False' )
OLITestLib.write_config_file(config)
# delete all remote and local testfolders
OLITestLib.delete_remote_testfolders()
OLITestLib.delete_maildir('')
OLITestLib.create_maildir('INBOX.OLItest')
code, res = OLITestLib.run_OLI()
#logging.warn("%s %s "% (code, res))
self.assertEqual(res, "")
OLITestLib.delete_maildir('INBOX.OLItest')
code, res = OLITestLib.run_OLI()
boxes, mails = OLITestLib.count_maildir_mails('')
self.assertTrue((boxes, mails)==(0,0), msg="Expected 0 folders and 0 "
"mails, but sync led to {} folders and {} mails".format(
boxes, mails))