tobias/docker-offlineimap
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branch 'next'

Browse Source
This commit is contained in:
Nicolas Sebrecht 2011-09-12 20:13:08 +02:00
commit e5a26dcfd8
31 changed files with 1211 additions and 930 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
Changelog.draft.rst
View File

@ -19,8 +19,6 @@ Changes
Bug Fixes
---------
Pending for the next major release
==================================

40
Changelog.rst
View File

@ -12,6 +12,45 @@ ChangeLog
releases announces.
OfflineIMAP v6.3.5-rc1 (2011-09-12)
===================================
Notes
-----
Idle feature and SQLite backend leave the experimental stage! ,-)
New Features
------------
* When a message upload/download fails, we do not abort the whole folder
synchronization, but only skip that message, informing the user at the
end of the sync run.
* If you connect via ssl and 'cert_fingerprint' is configured, we check
that the server certificate is actually known and identical by
comparing the stored sha1 fingerprint with the current one.
Changes
-------
* Refactor our IMAPServer class. Background work without user-visible
changes.
* Remove the configurability of the Blinkenlights statuschar. It
cluttered the main configuration file for little gain.
* Updated bundled imaplib2 to version 2.28.
Bug Fixes
---------
* We protect more robustly against asking for inexistent messages from the
IMAP server, when someone else deletes or moves messages while we sync.
* Selecting inexistent folders specified in folderincludes now throws
nice errors and continues to sync with all other folders rather than
exiting offlineimap with a traceback.
OfflineIMAP v6.3.4 (2011-08-10)
===============================
@ -26,6 +65,7 @@ Changes
* Handle when UID can't be found on saved messages.
OfflineIMAP v6.3.4-rc4 (2011-07-27)
===================================

385
docs/MANUAL.rst
View File

@ -14,39 +14,44 @@ Powerful IMAP/Maildir synchronization and reader support
.. TODO: :Manual group:
SYNOPSIS
========
offlineimap [-h|--help]
offlineimap [OPTIONS]
| -1
| -P profiledir
| -a accountlist
| -c configfile
| -d debugtype[,...]
| -f foldername[,...]
| -k [section:]option=value
| -l filename
| -o
| -u interface
DESCRIPTION
===========
Most configuration is done via the configuration file. Nevertheless, there are
a few command-line options that you may set for OfflineIMAP.
OfflineImap operates on a REMOTE and a LOCAL repository and synchronizes
emails between them, so that you can read the same mailbox from multiple
computers. The REMOTE repository is some IMAP server, while LOCAL can be
either a local Maildir or another IMAP server.
Missing folders will be automatically created on the LOCAL side, however
NO folders will currently be created on the REMOTE repository
automatically (it will sync your emails from local folders if
corresponding REMOTE folders already exist).
Configuring OfflineImap in basic mode is quite easy, however it provides
an amazing amount of flexibility for those with special needs. You can
specify the number of connections to your IMAP server, use arbitrary
python functions (including regular expressions) to limit the number of
folders being synchronized. You can transpose folder names between
repositories using any python function, to mangle and modify folder
names on the LOCAL repository. There are six different ways to hand the
IMAP password to OfflineImap from console input, specifying in the
configuration file, .netrc support, specifying in a separate file, to
using arbitrary python functions that somehow return the
password. Finally, you can use IMAPs IDLE infrastructure to always keep
a connection to your IMAP server open and immediately be notified (and
synchronized) when a new mail arrives (aka Push mail).
Most configuration is done via the configuration file. However, any setting can also be overriden by command line options handed to OfflineIMAP.
OfflineImap is well suited to be frequently invoked by cron jobs, or can run in daemon mode to periodically check your email (however, it will exit in some error situations).
Check out the `Use Cases`_ section for some example configurations.
OPTIONS
=======
-1 Disable most multithreading operations
Use solely a single-connection sync. This effectively sets the
@ -152,8 +157,7 @@ Blinkenlights
---------------
Blinkenlights is an interface designed to be sleek, fun to watch, and
informative of the overall picture of what OfflineIMAP is doing. I consider it
to be the best general-purpose interface in OfflineIMAP.
informative of the overall picture of what OfflineIMAP is doing.
Blinkenlights contains a row of "LEDs" with command buttons and a log.
@ -230,30 +234,27 @@ English-speaking world. One version ran in its entirety as follows:
TTYUI
---------
TTYUI interface is for people running in basic, non-color terminals. It
prints out basic status messages and is generally friendly to use on a console
or xterm.
TTYUI interface is for people running in terminals. It prints out basic
status messages and is generally friendly to use on a console or xterm.
Basic
--------------------
------
Basic is designed for situations in which OfflineIMAP will be run
non-attended and the status of its execution will be logged. You might use it,
for instance, to have the system run automatically and e-mail you the results of
the synchronization. This user interface is not capable of reading a password
from the keyboard; account passwords must be specified using one of the
configuration file options.
non-attended and the status of its execution will be logged. This user
interface is not capable of reading a password from the keyboard;
account passwords must be specified using one of the configuration file
options.
Quiet
-----
Quiet is designed for non-attended running in situations where normal
status messages are not desired. It will output nothing except errors
and serious warnings. Like Basic, this user interface is not capable
of reading a password from the keyboard; account passwords must be
specified using one of the configuration file options.
It will output nothing except errors and serious warnings. Like Basic,
this user interface is not capable of reading a password from the
keyboard; account passwords must be specified using one of the
configuration file options.
MachineUI
---------
@ -262,8 +263,98 @@ MachineUI generates output in a machine-parsable format. It is designed
for other programs that will interface to OfflineIMAP.
Signals
=======
Synchronization Performance
===========================
By default, we use fairly conservative settings that are safe for
syncing but that might not be the best performing one. Once you got
everything set up and running, you might want to look into speeding up
your synchronization. Here are a couple of hints and tips on how to
achieve this.
1) Use maxconnections > 1. By default we only use one connection to an
IMAP server. Using 2 or even 3 speeds things up considerably in most
cases. This setting goes into the [Repository XXX] section.
2) Use folderfilters. The quickest sync is a sync that can ignore some
folders. I sort my inbox into monthly folders, and ignore every
folder that is more than 2-3 months old, this lets me only inspect a
fraction of my Mails on every sync. If you haven't done this yet, do
it :). See the folderfilter section the example offlineimap.conf.
3) The default status cache is a plain text file that will write out
the complete file for each single new message (or even changed flag)
to a temporary file. If you have plenty of files in a folder, this
is a few hundred kilo to megabytes for each mail and is bound to
make things slower. I recommend to use the sqlite backend for
that. See the status_backend = sqlite setting in the example
offlineimap.conf. You will need to have python-sqlite installed in
order to use this. This will save you plenty of disk activity. Do
note that the sqlite backend is still considered experimental as it
has only been included recently (although a loss of your status
cache should not be a tragedy as that file can be rebuild
automatically)
4) Use quick sync. A regular sync will request all flags and all UIDs
of all mails in each folder which takes quite some time. A 'quick'
sync only compares the number of messages in a folder on the IMAP
side (it will detect flag changes on the Maildir side of things
though). A quick sync on my smallish account will take 7 seconds
rather than 40 seconds. Eg, I run a cron script that does a regular
sync once a day, and does quick syncs (-q) only synchronizing the
"-f INBOX" in between.
5) Turn off fsync. In the [general] section you can set fsync to True
or False. If you want to play 110% safe and wait for all operations
to hit the disk before continueing, you can set this to True. If you
set it to False, you lose some of that safety, trading it for speed.
Security and SSL
================
Some words on OfflineImap and its use of SSL/TLS. By default, we will
connect using any method that openssl supports, that is SSLv2, SSLv3, or
TLSv1. Do note that SSLv2 is notoriously insecure and deprecated.
Unfortunately, python2 does not offer easy ways to disable SSLv2. It is
recommended you test your setup and make sure that the mail server does
not use an SSLv2 connection. Use e.g. "openssl s_client -host
mail.server -port 443" to find out the connection that is used by
default.
Certificate checking
--------------------
Unfortunately, by default we will not verify the certificate of an IMAP
TLS/SSL server we connect to, so connecting by SSL is no guarantee
against man-in-the-middle attacks. While verifying a server certificate
fingerprint is being planned, it is not implemented yet. There is
currently only one safe way to ensure that you connect to the correct
server in an encrypted manner: You can specify a 'sslcacertfile' setting
in your repository section of offlineimap.conf pointing to a file that
contains (among others) a CA Certificate in PEM format which validating
your server certificate. In this case, we will check that: 1) The server
SSL certificate is validated by the CA Certificate 2) The server host
name matches the SSL certificate 3) The server certificate is not past
its expiration date. The FAQ contains an entry on how to create your own
certificate and CA certificate.
StartTLS
--------
If you have not configured your account to connect via SSL anyway,
OfflineImap will still attempt to set up an SSL connection via the
STARTTLS function, in case the imap server supports it. Do note, that
there is no certificate or fingerprint checking involved at all, when
using STARTTLS (the underlying imaplib library does not support this
yet). This means that you will be protected against passively listening
eavesdroppers and they will not be able to see your password or email
contents. However, this will not protect you from active attacks, such
as Man-In-The-Middle attacks which cause you to connect to the wrong
server and pretend to be your mail server. DO NOT RELY ON STARTTLS AS A
SAFE CONNECTION GUARANTEEING THE AUTHENTICITY OF YOUR IMAP SERVER!
UNIX Signals
============
OfflineImap listens to the unix signals SIGUSR1 and SIGUSR2.
@ -310,7 +401,7 @@ KNOWN BUGS
storing messages. Such files can be written to windows partitions. But
you will probably loose compatibility with other programs trying to
read the same Maildir.
- Exclamation mark was choosed because of the note in
- Exclamation mark was chosen because of the note in
http://docs.python.org/library/mailbox.html
- If you have some messages already stored without this option, you will
have to re-sync them again
@ -322,92 +413,146 @@ KNOWN BUGS
- not available anymore since cygwin 1.7
Synchronization Performance
===========================
PITFALLS & ISSUES
=================
By default, we use fairly conservative settings that are good for
syncing but that might not be the best performing one. Once you got
everything set up and running, you might want to look into speeding up
your synchronization. Here are a couple of hints and tips on how to
achieve this.
Sharing a maildir with multiple IMAP servers
--------------------------------------------
1) Use maxconnections > 1. By default we only use one connection to an
IMAP server. Using 2 or even 3 speeds things up considerably in most
cases. This setting goes into the [Repository XXX] section.
Generally a word of caution mixing IMAP repositories on the same
Maildir root. You have to be careful that you *never* use the same
maildir folder for 2 IMAP servers. In the best case, the folder MD5
will be different, and you will get a loop where it will upload your
mails to both servers in turn (infinitely!) as it thinks you have
placed new mails in the local Maildir. In the worst case, the MD5 is
the same (likely) and mail UIDs overlap (likely too!) and it will fail to
sync some mails as it thinks they are already existent.
2) Use folderfilters. The quickest sync is a sync that can ignore some
folders. I sort my inbox into monthly folders, and ignore every
folder that is more than 2-3 months old, this lets me only inspect a
fraction of my Mails on every sync. If you haven't done this yet, do
it :). See the folderfilter section the example offlineimap.conf.
I would create a new local Maildir Repository for the Personal Gmail and
use a different root to be on the safe side here. You could e.g. use
`~/mail/Pro` as Maildir root for the ProGmail and
`~/mail/Personal` as root for the personal one.
3) The default status cache is a plain text file that will write out
the complete file for each single new message (or even changed flag)
to a temporary file. If you have plenty of files in a folder, this
is a few hundred kilo to megabytes for each mail and is bound to
make things slower. I recommend to use the sqlite backend for
that. See the status_backend = sqlite setting in the example
offlineimap.conf. You will need to have python-sqlite installed in
order to use this. This will save you plenty of disk activity. Do
note that the sqlite backend is still considered experimental as it
has only been included recently (although a loss of your status
cache should not be a tragedy as that file can be rebuild
automatically)
If you then point your local mutt, or whatever MUA you use to `~/mail/`
as root, it should still recognize all folders. (see the 2 IMAP setup
in the `Use Cases`_ section.
4) Use quick sync. A regular sync will request all flags and all UIDs
of all mails in each folder which takes quite some time. A 'quick'
sync only compares the number of messages in a folder on the IMAP
side (it will detect flag changes on the Maildir side of things
though). A quick sync on my smallish account will take 7 seconds
rather than 40 seconds. Eg, I run a cron script that does a regular
sync once a day, and does quick syncs inbetween.
USE CASES
=========
5) Turn off fsync. In the [general] section you can set fsync to True
or False. If you want to play 110% safe and wait for all operations
to hit the disk before continueing, you can set this to True. If you
set it to False, you lose some of that safety trading it for speed.
Sync from GMail to another IMAP server
--------------------------------------
Security and SSL
================
This is an example of a setup where "TheOtherImap" requires all folders to be under INBOX::
Some words on OfflineImap and its use of SSL/TLS. By default, we will
connect using any method that openssl supports, that is SSLv2, SSLv3, or
TLSv1. Do note that SSLv2 is notoriously insecure and deprecated.
Unfortunately, python2 does not offer easy ways to disable SSLv2. It is
recommended you test your setup and make sure that the mail server does
not use an SSLv2 connection. Use e.g. "openssl s_client -host
mail.server -port 443" to find out the connection that is used by
default.
[Repository Gmailserver-foo]
#This is the remote repository
type = Gmail
remotepass = XXX
remoteuser = XXX
# The below will put all GMAIL folders as sub-folders of the 'local' INBOX,
# assuming that your path separator on 'local' is a dot.
nametrans = lambda x: 'INBOX.' + x
[Repository TheOtherImap]
#This is the 'local' repository
type = IMAP
remotehost = XXX
remotepass = XXX
remoteuser = XXX
#Do not use nametrans here.
Certificate checking
^^^^^^^^^^^^^^^^^^^^
Selecting only a few folders to sync
------------------------------------
Add this to the remote gmail repository section to only sync mails which are in a certain folder::
Unfortunately, by default we will not verify the certificate of an IMAP
TLS/SSL server we connect to, so connecting by SSL is no guarantee
against man-in-the-middle attacks. While verifying a server certificate
fingerprint is being planned, it is not implemented yet. There is
currently only one safe way to ensure that you connect to the correct
server in an encrypted manner: You can specify a 'sslcacertfile' setting
in your repository section of offlineimap.conf pointing to a file that
contains (among others) a CA Certificate in PEM format which validating
your server certificate. In this case, we will check that: 1) The server
SSL certificate is validated by the CA Certificate 2) The server host
name matches the SSL certificate 3) The server certificate is not past
its expiration date. The FAQ contains an entry on how to create your own
certificate and CA certificate.
folderfilter = lambda folder: folder.startswith('MyLabel')
StartTLS
^^^^^^^^
To only get the All Mail folder from a Gmail account, you would e.g. do::
If you have not configured your account to connect via SSL anyway,
OfflineImap will still attempt to set up an SSL connection via the
STARTTLS function, in case the imap server supports it. Do note, that
there is no certificate or fingerprint checking involved at all, when
using STARTTLS (the underlying imaplib library does not support this
yet). This means that you will be protected against passively listening
eavesdroppers and they will not be able to see your password or email
contents. However, this will not protect you from active attacks, such
as Man-In-The-Middle attacks which cause you to connect to the wrong
server and pretend to be your mail server. DO NOT RELY ON STARTTLS AS A
SAFE CONNECTION GUARANTEEING THE AUTHENTICITY OF YOUR IMAP SERVER!
=======
folderfilter = lambda folder: folder.startswith('[Gmail]/All Mail')
Another nametrans transpose example
-----------------------------------
Put everything in a GMX. subfolder except for the boxes INBOX, Draft, and Sent which should keep the same name::
nametrans: lambda folder: folder if folder in ['INBOX', 'Drafts', 'Sent'] \
else re.sub(r'^', r'GMX.', folder)
2 IMAP using name translations
------------------------------
Synchronizing 2 IMAP accounts to local Maildirs that are "next to each other", so that mutt can work on both. Full email setup described by Thomas Kahle at `http://dev.gentoo.org/~tomka/mail.html`_
offlineimap.conf::
[general]
accounts = acc1, acc2
maxsyncaccounts = 2
ui = ttyui
pythonfile=~/bin/offlineimap-helpers.py
socktimeout = 90
[Account acc1]
localrepository = acc1local
remoterepository = acc1remote
autorefresh = 2
[Account acc2]
localrepository = acc2local
remoterepository = acc2remote
autorefresh = 4
[Repository acc1local]
type = Maildir
localfolders = ~/Mail/acc1
[Repository acc2local]
type = Maildir
localfolders = ~/Mail/acc2
[Repository acc1remote]
type = IMAP
remotehost = imap.acc1.com
remoteusereval = get_username("imap.acc1.net")
remotepasseval = get_password("imap.acc1.net")
nametrans = oimaptransfolder_acc1
ssl = yes
maxconnections = 2
# Folders to get:
folderfilter = lambda foldername: foldername in [
'INBOX', 'Drafts', 'Sent', 'archiv']
[Repository acc2remote]
type = IMAP
remotehost = imap.acc2.net
remoteusereval = get_username("imap.acc2.net")
remotepasseval = get_password("imap.acc2.net")
nametrans = oimaptransfolder_acc2
ssl = yes
maxconnections = 2
One of the coolest things about offlineimap is that you can inject arbitrary python code. The file specified with::
pythonfile=~/bin/offlineimap-helpers.py
contains python functions that I used for two purposes: Fetching passwords from the gnome-keyring and translating folder names on the server to local foldernames. The python file should contain all the functions that are called here. get_username and get_password are part of the interaction with gnome-keyring and not printed here. Find them in the example file that is in the tarball or here. The folderfilter is a lambda term that, well, filters which folders to get. `oimaptransfolder_acc2` translates remote folders into local folders with a very simple logic. The `INBOX` folder will simply have the same name as the account while any other folder will have the account name and a dot as a prefix. offlineimap handles the renaming correctly in both directions::
import re
def oimaptransfolder_acc1(foldername):
if(foldername == "INBOX"):
retval = "acc1"
else:
retval = "acc1." + foldername
retval = re.sub("/", ".", retval)
return retval
def oimaptransfolder_acc2(foldername):
if(foldername == "INBOX"):
retval = "acc2"
else:
retval = "acc2." + foldername
retval = re.sub("/", ".", retval)
return retval

15
docs/dev-doc-src/index.rst
View File

@ -5,9 +5,9 @@
Welcome to :mod:`offlineimaps`'s documentation
==============================================
The :mod:`offlineimap` module provides the user interface for synchronization between IMAP servers and MailDirs or between IMAP servers. The homepage containing the source code repository can be found at the `offlineimap homepage <http://offlineimap.org>`_.
The :mod:`offlineimap` module provides the user interface for synchronization between IMAP servers and MailDirs or between IMAP servers. The homepage containing the source code repository can be found at the `offlineimap homepage <http://offlineimap.org>`_. The following provides the developer documentation for those who are interested in modifying the source code or otherwise peek into the OfflineImap internals. End users might want to check the MANUAL, our INSTALLation instructions, and the FAQ.
Within :mod:`offlineimap`, the classes :class:`OfflineImap` provides the high-level functionality. The rest of the classes should usually not needed to be touched by the user. A folder is represented by a :class:`offlineimap.folder.Base.BaseFolder` or any derivative :mod:`offlineimap.folder`.
Within :mod:`offlineimap`, the classes :class:`OfflineImap` provides the high-level functionality. The rest of the classes should usually not needed to be touched by the user. Email repositories are represented by a :class:`offlineimap.repository.Base.BaseRepository` or derivatives (see :mod:`offlineimap.repository` for details). A folder within a repository is represented by a :class:`offlineimap.folder.Base.BaseFolder` or any derivative from :mod:`offlineimap.folder`.
.. moduleauthor:: John Goerzen, and many others. See AUTHORS and the git history for a full list.
@ -15,7 +15,7 @@ Within :mod:`offlineimap`, the classes :class:`OfflineImap` provides the high-le
This page contains the main API overview of OfflineImap |release|.
Notmuch can be imported as::
OfflineImap can be imported as::
from offlineimap import OfflineImap
@ -58,11 +58,12 @@ An :class:`accounts.Account` connects two email repositories that are to be sync
Contains the current :mod:`offlineimap.ui`, and can be used for logging etc.
:exc:`OfflineImapException` -- A Notmuch execution error
:exc:`OfflineImapError` -- A Notmuch execution error
--------------------------------------------------------
.. autoexception:: offlineimap.OfflineImapException
.. autoexception:: offlineimap.error.OfflineImapError
:members:
This execption inherits directly from :exc:`Exception` and is raised on errors during the offlineimap execution.
This execption inherits directly from :exc:`Exception` and is raised
on errors during the offlineimap execution. It has an attribute
`severity` that denotes the severity level of the error.

4
docs/dev-doc-src/repository.rst
View File

@ -25,8 +25,8 @@ on your configuration. So when you want to instanciate a new
:mod:`offlineimap.repository` -- Basic representation of a mail repository
--------------------------------------------------------------------------
:mod:`offlineimap.repository.Base.BaseRepository` -- Representation of a mail repository
------------------------------------------------------------------------------------------
.. autoclass:: offlineimap.repository.Base.BaseRepository
:members:
:inherited-members:

2
docs/dev-doc-src/ui.rst
View File

@ -1,4 +1,4 @@
:mod:`offlineimap.ui` -- A pluggable logging system
:mod:`offlineimap.ui` -- A flexible logging system
--------------------------------------------------------
.. currentmodule:: offlineimap.ui

84
offlineimap.conf
View File

@ -15,8 +15,26 @@
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
# This file documents all possible options and can be quite scary.
# Looking for a quick start? Take a look at offlineimap.conf.minimal.
# Settings support interpolation. This means values can contain python
# format strings which refer to other values in the same section, or
# values in a special DEFAULT section. This allows you for example to
# use common settings for multiple accounts:
#
# [Repository Gmail1]
# trashfolder: %(gmailtrashfolder)s
#
# [Repository Gmail2]
# trashfolder: %(gmailtrashfolder)s
#
# [DEFAULT]
# gmailtrashfolder = [Google Mail]/Papierkorb
#
# would set the trashfolder setting for your German gmail accounts.
##################################################
# General definitions
@ -141,10 +159,6 @@ footer = "\n"
# Note that this filter can be used only to further restrict mbnames
# to a subset of folders that pass the account's folderfilter.
[ui.Curses.Blinkenlights]
# Character used to indicate thread status.
statuschar = .
##################################################
# Accounts
@ -205,8 +219,7 @@ remoterepository = RemoteExample
# state in plain text files. On Repositories with large numbers of
# mails, the performance might not be optimal, as we write out the
# complete file for each change. Another new backend 'sqlite' is
# available which stores the status in sqlite databases. BE AWARE THIS
# IS EXPERIMENTAL STUFF.
# available which stores the status in sqlite databases.
#
# If you switch the backend, you may want to delete the old cache
# directory in ~/.offlineimap/Account-<account>/LocalStatus manually
@ -314,6 +327,16 @@ ssl = yes
# The certificate should be in PEM format.
# sslcacertfile = /path/to/cacertfile.crt
# If you connect via SSL/TLS (ssl=true) and you have no CA certificate
# specified, offlineimap will refuse to sync as it connects to a server
# with an unknown "fingerprint". If you are sure you connect to the
# correct server, you can then configure the presented server
# fingerprint here. OfflineImap will verify that the server fingerprint
# has not changed on each connect and refuse to connect otherwise.
# You can also configure this in addition to CA certificate validation
# above and it will check both ways. cert_fingerprint =
# <SHA1_of_server_certificate_here>
# Specify the port. If not specified, use a default port.
# remoteport = 993
@ -383,8 +406,8 @@ remoteuser = username
# holdconnectionopen - to be true
# keepalive - to be 29 minutes unless you specify otherwise
#
# This feature isn't complete and may well have problems. BE AWARE THIS
# IS EXPERIMENTAL STUFF. See the manual for more details.
# This feature isn't complete and may well have problems. See the manual
# for more details.
#
# This option should return a Python list. For example
#
@ -447,10 +470,12 @@ subscribedonly = no
#
# nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)
# You can specify which folders to sync. You can do it several ways.
# I'll provide some examples. The folderfilter operates on the
# *UNTRANSLATED* name, if you specify nametrans. It should return
# true if the folder is to be included; false otherwise.
# You can specify which folders to sync using the folderfilter
# setting. You can provide any python function (e.g. a lambda function)
# which will be invoked for each foldername. If the filter function
# returns True, the folder will be synced, if it returns False, it. The
# folderfilter operates on the *UNTRANSLATED* name (before any nametrans
# translation takes place).
#
# Example 1: synchronizing only INBOX and Sent.
#
@ -474,34 +499,17 @@ subscribedonly = no
# folderfilter = lambda foldername: foldername in
# ['INBOX', 'Sent Mail', 'Deleted Items',
# 'Received']
#
# FYI, you could also include every folder with:
#
# folderfilter = lambda foldername: 1
#
# And exclude every folder with:
#
# folderfilter = lambda foldername: 0
# You can specify folderincludes to include additional folders.
# It should return a Python list. This might be used to include a
# folder that was excluded by your folderfilter rule, to include a
# folder that your server does not specify with its LIST option, or
# to include a folder that is outside your basic reference. Some examples:
#
# To include debian.user and debian.personal:
#
# You can specify folderincludes to include additional folders. It
# should return a Python list. This might be used to include a folder
# that was excluded by your folderfilter rule, to include a folder that
# your server does not specify with its LIST option, or to include a
# folder that is outside your basic reference. The 'reference' value
# will not be prefixed to this folder name, even if you have specified
# one. For example:
# folderincludes = ['debian.user', 'debian.personal']
#
# To include your INBOX (UW IMAPd users will find this useful if they
# specify a reference):
#
# folderincludes = ['INBOX']
#
# To specify a long list:
#
# folderincludes = ['box1', 'box2', 'box3', 'box4',
# 'box5', 'box6']
# You can specify foldersort to determine how folders are sorted.
# This affects order of synchronization and mbnames. The expression

4
offlineimap/CustomConfig.py
View File

@ -15,11 +15,11 @@
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
from ConfigParser import ConfigParser
from ConfigParser import SafeConfigParser
from offlineimap.localeval import LocalEval
import os
class CustomConfigParser(ConfigParser):
class CustomConfigParser(SafeConfigParser):
def getdefault(self, section, option, default, *args, **kwargs):
"""Same as config.get, but returns the "default" option if there
is no such option specified."""

2
offlineimap/__init__.py
View File

@ -1,7 +1,7 @@
__all__ = ['OfflineImap']
__productname__ = 'OfflineIMAP'
__version__ = "6.3.4"
__version__ = "6.3.5-rc1"
__copyright__ = "Copyright 2002-2011 John Goerzen & contributors"
__author__ = "John Goerzen"
__author_email__= "john@complete.org"

25
offlineimap/accounts.py
View File

@ -22,6 +22,7 @@ from offlineimap.threadutil import InstanceLimitedThread
from subprocess import Popen, PIPE
from threading import Event
import os
from sys import exc_info
import traceback
def getaccountlist(customconfig):
@ -178,16 +179,16 @@ class SyncableAccount(Account):
except (KeyboardInterrupt, SystemExit):
raise
except OfflineImapError, e:
self.ui.warn(e.reason)
# Stop looping and bubble up Exception if needed.
if e.severity >= OfflineImapError.ERROR.REPO:
if looping:
looping -= 1
if e.severity >= OfflineImapError.ERROR.CRITICAL:
raise
except:
self.ui.warn("Error occured attempting to sync account "\
"'%s':\n%s"% (self, traceback.format_exc()))
self.ui.error(e, exc_info()[2])
except Exception, e:
self.ui.error(e, msg = "While attempting to sync "
"account %s:\n %s"% (self, traceback.format_exc()))
else:
# after success sync, reset the looping counter to 3
if self.refreshperiod:
@ -232,7 +233,7 @@ class SyncableAccount(Account):
# replicate the folderstructure from REMOTE to LOCAL
if not localrepos.getconf('readonly', False):
self.ui.syncfolders(remoterepos, localrepos)
remoterepos.syncfoldersto(localrepos, [statusrepos])
remoterepos.syncfoldersto(localrepos, statusrepos)
# iterate through all folders on the remote repo and sync
for remotefolder in remoterepos.getfolders():
@ -276,8 +277,10 @@ class SyncableAccount(Account):
r = p.communicate()
self.ui.callhook("Hook stdout: %s\nHook stderr:%s\n" % r)
self.ui.callhook("Hook return code: %d" % p.returncode)
except:
self.ui.warn("Exception occured while calling hook")
except (KeyboardInterrupt, SystemExit):
raise
except Exception, e:
self.ui.error(e, exc_info()[2], msg = "Calling hook")
def syncfolder(accountname, remoterepos, remotefolder, localrepos,
@ -366,9 +369,9 @@ def syncfolder(accountname, remoterepos, remotefolder, localrepos,
if e.severity > OfflineImapError.ERROR.FOLDER:
raise
else:
ui.warn("Aborting folder sync '%s' [acc: '%s']\nReason was: %s" %\
(localfolder.name, accountname, e.reason))
except:
ui.warn("ERROR in syncfolder for %s folder %s: %s" % \
ui.error(e, exc_info()[2], msg = "Aborting folder sync '%s' "
"[acc: '%s']" % (localfolder, accountname))
except Exception, e:
ui.error(e, msg = "ERROR in syncfolder for %s folder %s: %s" % \
(accountname,remotefolder.getvisiblename(),
traceback.format_exc()))

16
offlineimap/error.py
View File

@ -2,11 +2,15 @@ class OfflineImapError(Exception):
"""An Error during offlineimap synchronization"""
class ERROR:
"""Severity levels"""
MESSAGE = 0
FOLDER = 10
REPO = 20
CRITICAL = 30
"""Severity level of an Exception
* **MESSAGE**: Abort the current message, but continue with folder
* **FOLDER_RETRY**: Error syncing folder, but do retry
* **FOLDER**: Abort folder sync, but continue with next folder
* **REPO**: Abort repository sync, continue with next account
* **CRITICAL**: Immediately exit offlineimap
"""
MESSAGE, FOLDER_RETRY, FOLDER, REPO, CRITICAL = 0, 10, 15, 20, 30
def __init__(self, reason, severity, errcode=None):
"""
@ -17,7 +21,7 @@ class OfflineImapError(Exception):
message, but a ERROR.REPO occurs when the server is
offline.
:param errcode: optional number denoting a predefined error
:param errcode: optional number denoting a predefined error
situation (which let's us exit with a predefined exit
value). So far, no errcodes have been defined yet.

117
offlineimap/folder/Base.py
View File

@ -17,9 +17,15 @@
from offlineimap import threadutil
from offlineimap.ui import getglobalui
from offlineimap.error import OfflineImapError
import os.path
import re
from sys import exc_info
import traceback
try: # python 2.6 has set() built in
set
except NameError:
from sets import Set as set
class BaseFolder(object):
def __init__(self):
@ -70,11 +76,15 @@ class BaseFolder(object):
return self.getname()
def getfolderbasename(self):
foldername = self.getname()
foldername = foldername.replace(self.repository.getsep(), '.')
foldername = re.sub('/\.$', '/dot', foldername)
foldername = re.sub('^\.$', 'dot', foldername)
return foldername
"""Return base file name of file to store Status/UID info in"""
if not self.name:
basename = '.'
else: #avoid directory hierarchies and file names such as '/'
basename = self.name.replace('/', '.')
# replace with literal 'dot' if final path name is '.' as '.' is
# an invalid file name.
basename = re.sub('(^|\/)\.$','\\1dot', basename)
return basename
def isuidvalidityok(self):
"""Does the cached UID match the real UID
@ -183,12 +193,9 @@ class BaseFolder(object):
def addmessageflags(self, uid, flags):
"""Adds the specified flags to the message's flag set. If a given
flag is already present, it will not be duplicated."""
newflags = self.getmessageflags(uid)
for flag in flags:
if not flag in newflags:
newflags.append(flag)
newflags.sort()
flag is already present, it will not be duplicated.
:param flags: A set() of flags"""
newflags = self.getmessageflags(uid) | flags
self.savemessageflags(uid, newflags)
def addmessagesflags(self, uidlist, flags):
@ -198,11 +205,7 @@ class BaseFolder(object):
def deletemessageflags(self, uid, flags):
"""Removes each flag given from the message's flag set. If a given
flag is already removed, no action will be taken for that flag."""
newflags = self.getmessageflags(uid)
for flag in flags:
if flag in newflags:
newflags.remove(flag)
newflags.sort()
newflags = self.getmessageflags(uid) - flags
self.savemessageflags(uid, newflags)
def deletemessagesflags(self, uidlist, flags):
@ -229,10 +232,10 @@ class BaseFolder(object):
# synced to the status cache. This is only a problem with
# self.getmessage(). So, don't call self.getmessage unless
# really needed.
try:
if register: # output that we start a new thread
self.ui.registerthread(self.getaccountname())
if register: # output that we start a new thread
self.ui.registerthread(self.getaccountname())
try:
message = None
flags = self.getmessageflags(uid)
rtime = self.getmessagetime(uid)
@ -242,17 +245,17 @@ class BaseFolder(object):
statusfolder.savemessage(uid, None, flags, rtime)
return
self.ui.copyingmessage(uid, self, [dstfolder])
self.ui.copyingmessage(uid, self, dstfolder)
# If any of the destinations actually stores the message body,
# load it up.
if dstfolder.storesmessages():
message = self.getmessage(uid)
#Succeeded? -> IMAP actually assigned a UID. If newid
#remained negative, no server was willing to assign us an
#UID. If newid is 0, saving succeeded, but we could not
#retrieve the new UID. Ignore message in this case.
newuid = dstfolder.savemessage(uid, message, flags, rtime)
if newuid > 0:
if newuid != uid:
# Got new UID, change the local uid.
@ -264,6 +267,7 @@ class BaseFolder(object):
uid = newuid
# Save uploaded status in the statusfolder
statusfolder.savemessage(uid, message, flags, rtime)
elif newuid == 0:
# Message was stored to dstfolder, but we can't find it's UID
# This means we can't link current message to the one created
@ -273,18 +277,22 @@ class BaseFolder(object):
# IMAP servers ...
self.deletemessage(uid)
else:
raise UserWarning("Trying to save msg (uid %d) on folder "
raise OfflineImapError("Trying to save msg (uid %d) on folder "
"%s returned invalid uid %d" % \
(uid,
dstfolder.getvisiblename(),
newuid))
except (KeyboardInterrupt):
raise
except:
self.ui.warn("ERROR attempting to copy message " + str(uid) \
+ " for account " + self.getaccountname() + ":" \
+ traceback.format_exc())
raise
newuid),
OfflineImapError.ERROR.MESSAGE)
except OfflineImapError, e:
if e.severity > OfflineImapError.ERROR.MESSAGE:
raise # buble severe errors up
self.ui.error(e, exc_info()[2])
except Exception, e:
self.ui.error(e, "Copying message %s [acc: %s]:\n %s" %\
(uid, self.getaccountname(),
traceback.format_exc()))
raise #raise on unknown errors, so we can fix those
def syncmessagesto_copy(self, dstfolder, statusfolder):
"""Pass1: Copy locally existing messages not on the other side
@ -303,20 +311,21 @@ class BaseFolder(object):
statusfolder.uidexists(uid),
self.getmessageuidlist())
for uid in copylist:
# exceptions are caught in copymessageto()
if self.suggeststhreads():
self.waitforthread()
thread = threadutil.InstanceLimitedThread(\
self.getcopyinstancelimit(),
target = self.copymessageto,
name = "Copy message %d from %s" % (uid,
self.getvisiblename()),
self.getvisiblename()),
args = (uid, dstfolder, statusfolder))
thread.setDaemon(1)
thread.start()
threads.append(thread)
else:
self.copymessageto(uid, dstfolder, statusfolder, register = 0)
self.copymessageto(uid, dstfolder, statusfolder,
register = 0)
for thread in threads:
thread.join()
@ -350,8 +359,8 @@ class BaseFolder(object):
addflaglist = {}
delflaglist = {}
for uid in self.getmessageuidlist():
# Ignore messages with negative UIDs missed by pass 1
# also don't do anything if the message has been deleted remotely
# Ignore messages with negative UIDs missed by pass 1 and
# don't do anything if the message has been deleted remotely
if uid < 0 or not dstfolder.uidexists(uid):
continue
@ -359,30 +368,31 @@ class BaseFolder(object):
statusflags = statusfolder.getmessageflags(uid)
#if we could not get message flags from LocalStatus, assume empty.
if statusflags is None:
statusflags = []
addflags = [x for x in selfflags if x not in statusflags]
statusflags = set()
addflags = selfflags - statusflags
delflags = statusflags - selfflags
for flag in addflags:
if not flag in addflaglist:
addflaglist[flag] = []
addflaglist[flag].append(uid)
delflags = [x for x in statusflags if x not in selfflags]
for flag in delflags:
if not flag in delflaglist:
delflaglist[flag] = []
delflaglist[flag].append(uid)
for flag in addflaglist.keys():
self.ui.addingflags(addflaglist[flag], flag, dstfolder)
dstfolder.addmessagesflags(addflaglist[flag], [flag])
statusfolder.addmessagesflags(addflaglist[flag], [flag])
for flag in delflaglist.keys():
self.ui.deletingflags(delflaglist[flag], flag, dstfolder)
dstfolder.deletemessagesflags(delflaglist[flag], [flag])
statusfolder.deletemessagesflags(delflaglist[flag], [flag])
for flag, uids in addflaglist.items():
self.ui.addingflags(uids, flag, dstfolder)
dstfolder.addmessagesflags(uids, set(flag))
statusfolder.addmessagesflags(uids, set(flag))
for flag,uids in delflaglist.items():
self.ui.deletingflags(uids, flag, dstfolder)
dstfolder.deletemessagesflags(uids, set(flag))
statusfolder.deletemessagesflags(uids, set(flag))
def syncmessagesto(self, dstfolder, statusfolder):
"""Syncs messages in this folder to the destination dstfolder.
@ -421,8 +431,11 @@ class BaseFolder(object):
action(dstfolder, statusfolder)
except (KeyboardInterrupt):
raise
except:
self.ui.warn("ERROR attempting to sync flags " \
+ "for account " + self.getaccountname() \
+ ":" + traceback.format_exc())
raise
except OfflineImapError, e:
if e.severity > OfflineImapError.ERROR.FOLDER:
raise
self.ui.error(e, exc_info()[2])
except Exception, e:
self.ui.error(e, exc_info()[2], "Syncing folder %s [acc: %s]" %\
(self, self.getaccountname()))
raise # raise unknown Exceptions so we can fix them

59
offlineimap/folder/Gmail.py
View File

@ -21,7 +21,6 @@
from IMAP import IMAPFolder
from offlineimap import imaputil
from copy import copy
class GmailFolder(IMAPFolder):
@ -45,7 +44,7 @@ class GmailFolder(IMAPFolder):
def deletemessages_noconvert(self, uidlist):
uidlist = [uid for uid in uidlist if uid in self.messagelist]
if not len(uidlist):
return
return
if self.realdelete and not (self.getname() in self.real_delete_folders):
# IMAP expunge is just "remove label" in this folder,
@ -55,7 +54,7 @@ class GmailFolder(IMAPFolder):
try:
imapobj.select(self.getfullname())
result = imapobj.uid('copy',
imaputil.listjoin(uidlist),
imaputil.uid_sequence(uidlist),
self.trash_folder)
assert result[0] == 'OK', \
"Bad IMAPlib result: %s" % result[0]
@ -65,57 +64,3 @@ class GmailFolder(IMAPFolder):
del self.messagelist[uid]
else:
IMAPFolder.deletemessages_noconvert(self, uidlist)
def processmessagesflags(self, operation, uidlist, flags):
# XXX: the imapobj.myrights(...) calls dies with an error
# report from Gmail server stating that IMAP command
# 'MYRIGHTS' is not implemented. So, this
# `processmessagesflags` is just a copy from `IMAPFolder`,
# with the references to `imapobj.myrights()` deleted This
# shouldn't hurt, however, Gmail users always have full
# control over all their mailboxes (apparently).
if len(uidlist) > 101:
# Hack for those IMAP ervers with a limited line length
self.processmessagesflags(operation, uidlist[:100], flags)
self.processmessagesflags(operation, uidlist[100:], flags)
return
imapobj = self.imapserver.acquireconnection()
try:
imapobj.select(self.getfullname())
r = imapobj.uid('store',
imaputil.listjoin(uidlist),
operation + 'FLAGS',
imaputil.flagsmaildir2imap(flags))
assert r[0] == 'OK', 'Error with store: ' + '. '.join(r[1])
r = r[1]
finally:
self.imapserver.releaseconnection(imapobj)
needupdate = copy(uidlist)
for result in r:
if result == None:
# Compensate for servers that don't return anything from
# STORE.
continue
attributehash = imaputil.flags2hash(imaputil.imapsplit(result)[1])
if not ('UID' in attributehash and 'FLAGS' in attributehash):
# Compensate for servers that don't return a UID attribute.
continue
flags = attributehash['FLAGS']
uid = long(attributehash['UID'])
self.messagelist[uid]['flags'] = imaputil.flagsimap2maildir(flags)
try:
needupdate.remove(uid)
except ValueError: # Let it slide if it's not in the list
pass
for uid in needupdate:
if operation == '+':
for flag in flags:
if not flag in self.messagelist[uid]['flags']:
self.messagelist[uid]['flags'].append(flag)
self.messagelist[uid]['flags'].sort()
elif operation == '-':
for flag in flags:
if flag in self.messagelist[uid]['flags']:
self.messagelist[uid]['flags'].remove(flag)

327
offlineimap/folder/IMAP.py
View File

@ -21,9 +21,15 @@ import random
import binascii
import re
import time
from copy import copy
from sys import exc_info
from Base import BaseFolder
from offlineimap import imaputil, imaplibutil, OfflineImapError
from offlineimap.imaplib2 import MonthNames
try: # python 2.6 has set() built in
set
except NameError:
from sets import Set as set
class IMAPFolder(BaseFolder):
def __init__(self, imapserver, name, visiblename, accountname, repository):
@ -105,78 +111,75 @@ class IMAPFolder(BaseFolder):
self.imapserver.releaseconnection(imapobj)
return False
# TODO: Make this so that it can define a date that would be the oldest messages etc.
def cachemessagelist(self):
imapobj = self.imapserver.acquireconnection()
maxage = self.config.getdefaultint("Account %s" % self.accountname,
"maxage", -1)
maxsize = self.config.getdefaultint("Account %s" % self.accountname,
"maxsize", -1)
self.messagelist = {}
imapobj = self.imapserver.acquireconnection()
try:
# Primes untagged_responses
imaptype, imapdata = imapobj.select(self.getfullname(), readonly = 1, force = 1)
maxage = self.config.getdefaultint("Account " + self.accountname, "maxage", -1)
maxsize = self.config.getdefaultint("Account " + self.accountname, "maxsize", -1)
res_type, imapdata = imapobj.select(self.getfullname(), True)
if imapdata == [None] or imapdata[0] == '0':
# Empty folder, no need to populate message list
return
# By default examine all UIDs in this folder
msgsToFetch = '1:*'
if (maxage != -1) | (maxsize != -1):
try:
search_condition = "(";
search_cond = "(";
if(maxage != -1):
#find out what the oldest message is that we should look at
oldest_time_struct = time.gmtime(time.time() - (60*60*24*maxage))
if(maxage != -1):
#find out what the oldest message is that we should look at
oldest_struct = time.gmtime(time.time() - (60*60*24*maxage))
if oldest_struct[0] < 1900:
raise OfflineImapError("maxage setting led to year %d. "
"Abort syncing." % oldest_struct[0],
OfflineImapError.ERROR.REPO)
search_cond += "SINCE %02d-%s-%d" % (
oldest_struct[2],
MonthNames[oldest_struct[1]],
oldest_struct[0])
#format this manually - otherwise locales could cause problems
monthnames_standard = ["Jan", "Feb", "Mar", "Apr", "May", \
"Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]
if(maxsize != -1):
if(maxage != -1): # There are two conditions, add space
search_cond += " "
search_cond += "SMALLER %d" % maxsize
our_monthname = monthnames_standard[oldest_time_struct[1]-1]
daystr = "%(day)02d" % {'day' : oldest_time_struct[2]}
date_search_str = "SINCE " + daystr + "-" + our_monthname \
+ "-" + str(oldest_time_struct[0])
search_cond += ")"
search_condition += date_search_str
res_type, res_data = imapobj.search(None, search_cond)
if res_type != 'OK':
raise OfflineImapError("SEARCH in folder [%s]%s failed. "
"Search string was '%s'. Server responded '[%s] %s'" % (
self.getrepository(), self,
search_cond, res_type, res_data),
OfflineImapError.ERROR.FOLDER)
if(maxsize != -1):
if(maxage != -1): #There are two conditions - add a space
search_condition += " "
# Result UIDs are seperated by space, coalesce into ranges
msgsToFetch = imaputil.uid_sequence(res_data[0].split())
if not msgsToFetch:
return # No messages to sync
search_condition += "SMALLER " + self.config.getdefault("Account " + self.accountname, "maxsize", -1)
search_condition += ")"
searchresult = imapobj.search(None, search_condition)
#result would come back seperated by space - to change into a fetch
#statement we need to change space to comma
messagesToFetch = searchresult[1][0].replace(" ", ",")
except KeyError:
return
if len(messagesToFetch) < 1:
# No messages; return
return
else:
# 1. Some mail servers do not return an EXISTS response
# if the folder is empty. 2. ZIMBRA servers can return
# multiple EXISTS replies in the form 500, 1000, 1500,
# 1623 so check for potentially multiple replies.
if imapdata == [None]:
return
maxmsgid = 0
for msgid in imapdata:
maxmsgid = max(long(msgid), maxmsgid)
if maxmsgid < 1:
#no messages; return
return
messagesToFetch = '1:%d' % maxmsgid;
# Now, get the flags and UIDs for these.
# We could conceivably get rid of maxmsgid and just say
# '1:*' here.
response = imapobj.fetch(messagesToFetch, '(FLAGS UID)')[1]
# Get the flags and UIDs for these. single-quotes prevent
# imaplib2 from quoting the sequence.
res_type, response = imapobj.fetch("'%s'" % msgsToFetch,
'(FLAGS UID)')
if res_type != 'OK':
raise OfflineImapError("FETCHING UIDs in folder [%s]%s failed. "
"Server responded '[%s] %s'" % (
self.getrepository(), self,
res_type, response),
OfflineImapError.ERROR.FOLDER)
finally:
self.imapserver.releaseconnection(imapobj)
for messagestr in response:
# Discard the message number.
# looks like: '1 (FLAGS (\\Seen Old) UID 4807)' or None if no msg
# Discard initial message number.
if messagestr == None:
continue
messagestr = messagestr.split(' ', 1)[1]
options = imaputil.flags2hash(messagestr)
if not options.has_key('UID'):
@ -201,13 +204,27 @@ class IMAPFolder(BaseFolder):
"""
imapobj = self.imapserver.acquireconnection()
try:
imapobj.select(self.getfullname(), readonly = 1)
res_type, data = imapobj.uid('fetch', str(uid), '(BODY.PEEK[])')
fails_left = 2 # retry on dropped connection
while fails_left:
try:
imapobj.select(self.getfullname(), readonly = 1)
res_type, data = imapobj.uid('fetch', str(uid),
'(BODY.PEEK[])')
fails_left = 0
except imapobj.abort(), e:
# Release dropped connection, and get a new one
self.imapserver.releaseconnection(imapobj)
imapobj = self.imapserver.acquireconnection()
self.ui.error(e, exc_info()[2])
fails_left -= 1
if not fails_left:
raise e
if data == [None] or res_type != 'OK':
#IMAP server says bad request or UID does not exist
severity = OfflineImapError.ERROR.MESSAGE
reason = "IMAP server '%s' responded with '%s' to fetching "\
"message UID '%d'" % (self.getrepository(), res_type, uid)
reason = "IMAP server '%s' failed to fetch message UID '%d'."\
"Server responded: %s %s" % (self.getrepository(), uid,
res_type, data)
if data == [None]:
#IMAP server did not find a message with this UID
reason = "IMAP server '%s' does not have a message "\
@ -216,16 +233,6 @@ class IMAPFolder(BaseFolder):
# data looks now e.g. [('320 (UID 17061 BODY[]
# {2565}','msgbody....')] we only asked for one message,
# and that msg is in data[0]. msbody is in [0][1]
#NB & TODO: When the message on the IMAP server has been
#deleted in the mean time, it will respond with an 'OK'
#res_type, but it will simply not send any data. This will
#lead to a crash in the below line. We need urgently to
#detect this, protect from this and need to think about what
#to return in this case. Probably returning `None` in this
#case would be good. But we need to make sure that all
#Backends behave the same, and that we actually check the
#return value and behave accordingly.
data = data[0][1].replace("\r\n", "\n")
if len(data)>200:
@ -317,6 +324,74 @@ class IMAPFolder(BaseFolder):
matchinguids.sort()
return long(matchinguids[0])
def savemessage_fetchheaders(self, imapobj, headername, headervalue):
""" We fetch all new mail headers and search for the right
X-OfflineImap line by hand. The response from the server has form:
(
'OK',
[
(
'185 (RFC822.HEADER {1789}',
'... mail headers ...'
),
' UID 2444)',
(
'186 (RFC822.HEADER {1789}',
'... 2nd mail headers ...'
),
' UID 2445)'
]
)
We need to locate the UID just after mail headers containing our
X-OfflineIMAP line.
Returns UID when found, 0 when not found.
"""
self.ui.debug('imap', 'savemessage_fetchheaders called for %s: %s' % \
(headername, headervalue))
# run "fetch X:* rfc822.header"
# since we stored the mail we are looking for just recently, it would
# not be optimal to fetch all messages. So we'll find highest message
# UID in our local messagelist and search from there (exactly from
# UID+1). That works because UIDs are guaranteed to be unique and
# ascending.
if self.getmessagelist():
start = 1+max(self.getmessagelist().keys())
else:
# Folder was empty - start from 1
start = 1
# Imaplib quotes all parameters of a string type. That must not happen
# with the range X:*. So we use bytearray to stop imaplib from getting
# in our way
result = imapobj.uid('FETCH', bytearray('%d:*' % start), 'rfc822.header')
if result[0] != 'OK':
raise OfflineImapError('Error fetching mail headers: ' + '. '.join(result[1]),
OfflineImapError.ERROR.MESSAGE)
result = result[1]
found = 0
for item in result:
if found == 0 and type(item) == type( () ):
# Walk just tuples
if re.search("(?:^|\\r|\\n)%s:\s*%s(?:\\r|\\n)" % (headername, headervalue),
item[1], flags=re.IGNORECASE):
found = 1
elif found == 1:
if type(item) == type (""):
uid = re.search("UID\s+(\d+)", item, flags=re.IGNORECASE)
if uid:
return int(uid.group(1))
else:
self.ui.warn("Can't parse FETCH response, can't find UID: %s", result.__repr__())
else:
self.ui.warn("Can't parse FETCH response, we awaited string: %s", result.__repr__())
return 0
def getmessageinternaldate(self, content, rtime=None):
"""Parses mail and returns an INTERNALDATE string
@ -420,46 +495,64 @@ class IMAPFolder(BaseFolder):
self.savemessageflags(uid, flags)
return uid
imapobj = self.imapserver.acquireconnection()
try:
imapobj = self.imapserver.acquireconnection()
success = False # succeeded in APPENDING?
while not success:
try:
imapobj.select(self.getfullname()) # Needed for search and making the box READ-WRITE
except imapobj.readonly:
# readonly exception. Return original uid to notify that
# we did not save the message. (see savemessage in Base.py)
self.ui.msgtoreadonly(self, uid, content, flags)
return uid
# UIDPLUS extension provides us with an APPENDUID response.
use_uidplus = 'UIDPLUS' in imapobj.capabilities
# UIDPLUS extension provides us with an APPENDUID response to our append()
use_uidplus = 'UIDPLUS' in imapobj.capabilities
# get the date of the message, so we can pass it to the server.
date = self.getmessageinternaldate(content, rtime)
content = re.sub("(?<!\r)\n", "\r\n", content)
# get the date of the message file, so we can pass it to the server.
date = self.getmessageinternaldate(content, rtime)
content = re.sub("(?<!\r)\n", "\r\n", content)
if not use_uidplus:
# insert a random unique header that we can fetch later
(headername, headervalue) = self.generate_randomheader(
content)
self.ui.debug('imap', 'savemessage: header is: %s: %s' %\
(headername, headervalue))
content = self.savemessage_addheader(content, headername,
headervalue)
if len(content)>200:
dbg_output = "%s...%s" % (content[:150], content[-50:])
else:
dbg_output = content
self.ui.debug('imap', "savemessage: date: %s, content: '%s'" %
(date, dbg_output))
if not use_uidplus:
# insert a random unique header that we can fetch later
(headername, headervalue) = self.generate_randomheader(content)
self.ui.debug('imap', 'savemessage: new header is: %s: %s' % \
(headername, headervalue))
content = self.savemessage_addheader(content, headername,
headervalue)
if len(content)>200:
dbg_output = "%s...%s" % (content[:150],
content[-50:])
else:
dbg_output = content
try:
# Select folder for append and make the box READ-WRITE
imapobj.select(self.getfullname())
except imapobj.readonly:
# readonly exception. Return original uid to notify that
# we did not save the message. (see savemessage in Base.py)
self.ui.msgtoreadonly(self, uid, content, flags)
return uid
self.ui.debug('imap', "savemessage: date: %s, content: '%s'" %
(date, dbg_output))
(typ,dat) = imapobj.append(self.getfullname(),
#Do the APPEND
try:
(typ, dat) = imapobj.append(self.getfullname(),
imaputil.flagsmaildir2imap(flags),
date, content)
assert(typ == 'OK')
# Checkpoint. Let it write out the messages, etc.
success = True
except imapobj.abort, e:
# connection has been reset, release connection and retry.
self.ui.error(e, exc_info()[2])
self.imapserver.releaseconnection(imapobj, True)
imapobj = self.imapserver.acquireconnection()
except imapobj.error, e:
# If the server responds with 'BAD', append() raise()s directly.
# So we need to prepare a response ourselves.
typ, dat = 'BAD', str(e)
if typ != 'OK': #APPEND failed
raise OfflineImapError("Saving msg in folder '%s', repository "
"'%s' failed. Server reponded; %s %s\nMessage content was:"
" %s" % (self, self.getrepository(), typ, dat, dbg_output),
OfflineImapError.ERROR.MESSAGE)
# Checkpoint. Let it write out stuff, etc. Eg searches for
# just uploaded messages won't work if we don't do this.
(typ,dat) = imapobj.check()
assert(typ == 'OK')
@ -481,10 +574,15 @@ class IMAPFolder(BaseFolder):
headervalue)
# See docs for savemessage in Base.py for explanation of this and other return values
if uid == 0:
self.ui.debug('imap', 'savemessage: first attempt to get new UID failed. Going to run a NOOP and try again.')
self.ui.debug('imap', 'savemessage: first attempt to get new UID failed. \
Going to run a NOOP and try again.')
assert(imapobj.noop()[0] == 'OK')
uid = self.savemessage_searchforheader(imapobj, headername,
headervalue)
if uid == 0:
self.ui.debug('imap', 'savemessage: second attempt to get new UID failed. \
Going to try search headers manually')
uid = self.savemessage_fetchheaders(imapobj, headername, headervalue)
finally:
self.imapserver.releaseconnection(imapobj)
@ -549,7 +647,7 @@ class IMAPFolder(BaseFolder):
self.ui.flagstoreadonly(self, uidlist, flags)
return
r = imapobj.uid('store',
imaputil.listjoin(uidlist),
imaputil.uid_sequence(uidlist),
operation + 'FLAGS',
imaputil.flagsmaildir2imap(flags))
assert r[0] == 'OK', 'Error with store: ' + '. '.join(r[1])
@ -559,7 +657,7 @@ class IMAPFolder(BaseFolder):
# Some IMAP servers do not always return a result. Therefore,
# only update the ones that it talks about, and manually fix
# the others.
needupdate = copy(uidlist)
needupdate = list(uidlist)
for result in r:
if result == None:
# Compensate for servers that don't return anything from
@ -569,23 +667,18 @@ class IMAPFolder(BaseFolder):
if not ('UID' in attributehash and 'FLAGS' in attributehash):
# Compensate for servers that don't return a UID attribute.
continue
lflags = attributehash['FLAGS']
flagstr = attributehash['FLAGS']
uid = long(attributehash['UID'])
self.messagelist[uid]['flags'] = imaputil.flagsimap2maildir(lflags)
self.messagelist[uid]['flags'] = imaputil.flagsimap2maildir(flagstr)
try:
needupdate.remove(uid)
except ValueError: # Let it slide if it's not in the list
pass
for uid in needupdate:
if operation == '+':
for flag in flags:
if not flag in self.messagelist[uid]['flags']:
self.messagelist[uid]['flags'].append(flag)
self.messagelist[uid]['flags'].sort()
self.messagelist[uid]['flags'] |= flags
elif operation == '-':
for flag in flags:
if flag in self.messagelist[uid]['flags']:
self.messagelist[uid]['flags'].remove(flag)
self.messagelist[uid]['flags'] -= flags
def deletemessage(self, uid):
self.deletemessages_noconvert([uid])
@ -599,7 +692,7 @@ class IMAPFolder(BaseFolder):
if not len(uidlist):
return
self.addmessagesflags_noconvert(uidlist, ['T'])
self.addmessagesflags_noconvert(uidlist, set('T'))
imapobj = self.imapserver.acquireconnection()
try:
try:

17
offlineimap/folder/LocalStatus.py
View File

@ -1,6 +1,5 @@
# Local status cache virtual folder
# Copyright (C) 2002 - 2008 John Goerzen
# <jgoerzen@complete.org>
# Copyright (C) 2002 - 2011 John Goerzen & contributors
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@ -19,6 +18,10 @@
from Base import BaseFolder
import os
import threading
try: # python 2.6 has set() built in
set
except NameError:
from sets import Set as set
magicline = "OFFLINEIMAP LocalStatus CACHE DATA - DO NOT MODIFY - FORMAT 1"
@ -28,7 +31,7 @@ class LocalStatusFolder(BaseFolder):
self.root = root
self.sep = '.'
self.config = config
self.filename = repository.getfolderfilename(name)
self.filename = os.path.join(root, self.getfolderbasename())
self.messagelist = {}
self.repository = repository
self.savelock = threading.Lock()
@ -80,11 +83,12 @@ class LocalStatusFolder(BaseFolder):
try:
uid, flags = line.split(':')
uid = long(uid)
flags = set(flags)
except ValueError, e:
errstr = "Corrupt line '%s' in cache file '%s'" % (line, self.filename)
errstr = "Corrupt line '%s' in cache file '%s'" % \
(line, self.filename)
self.ui.warn(errstr)
raise ValueError(errstr)
flags = [x for x in flags]
self.messagelist[uid] = {'uid': uid, 'flags': flags}
file.close()
@ -95,8 +99,7 @@ class LocalStatusFolder(BaseFolder):
file.write(magicline + "\n")
for msg in self.messagelist.values():
flags = msg['flags']
flags.sort()
flags = ''.join(flags)
flags = ''.join(sorted(flags))
file.write("%s:%s\n" % (msg['uid'], flags))
file.flush()
if self.doautosave:

16
offlineimap/folder/LocalStatusSQLite.py
View File

@ -23,6 +23,11 @@ try:
except:
pass #fail only if needed later on, not on import
try: # python 2.6 has set() built in
set
except NameError:
from sets import Set as set
class LocalStatusSQLiteFolder(LocalStatusFolder):
"""LocalStatus backend implemented with an SQLite database
@ -106,7 +111,8 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
if hasattr(self, 'connection'):
self.connection.close() #close old connections first
self.connection = sqlite.connect(self.filename, check_same_thread = False)
self.connection = sqlite.connect(self.filename,
check_same_thread = False)
if from_ver == 0:
# from_ver==0: no db existent: plain text migration?
@ -115,7 +121,7 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
plaintextfilename = os.path.join(
self.repository.account.getaccountmeta(),
'LocalStatus',
re.sub('(^|\/)\.$','\\1dot', self.name))
self.getfolderbasename())
# MIGRATE from plaintext if needed
if os.path.exists(plaintextfilename):
self.ui._msg('Migrating LocalStatus cache from plain text '
@ -127,7 +133,6 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
for line in file.xreadlines():
uid, flags = line.strip().split(':')
uid = long(uid)
flags = list(flags)
flags = ''.join(sorted(flags))
data.append((uid,flags))
self.connection.executemany('INSERT INTO status (id,flags) VALUES (?,?)',
@ -167,7 +172,7 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
self.messagelist = {}
cursor = self.connection.execute('SELECT id,flags from status')
for row in cursor:
flags = [x for x in row[1]]
flags = set(row[1])
self.messagelist[row[0]] = {'uid': row[0], 'flags': flags}
def save(self):
@ -227,8 +232,7 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
def savemessageflags(self, uid, flags):
self.messagelist[uid] = {'uid': uid, 'flags': flags}
flags.sort()
flags = ''.join(flags)
flags = ''.join(sorted(flags))
self.sql_write('UPDATE status SET flags=? WHERE id=?',(flags,uid))
def deletemessages(self, uidlist):

35
offlineimap/folder/Maildir.py
View File

@ -28,6 +28,11 @@ try:
except ImportError:
from md5 import md5
try: # python 2.6 has set() built in
set
except NameError:
from sets import Set as set
from offlineimap import OfflineImapError
uidmatchre = re.compile(',U=(\d+)')
@ -128,7 +133,7 @@ class MaildirFolder(BaseFolder):
folderstr = ',FMD5=' + foldermd5
for dirannex in ['new', 'cur']:
fulldirname = os.path.join(self.getfullname(), dirannex)
files.extend(os.path.join(fulldirname, filename) for
files.extend(os.path.join(dirannex, filename) for
filename in os.listdir(fulldirname))
for file in files:
messagename = os.path.basename(file)
@ -146,10 +151,9 @@ class MaildirFolder(BaseFolder):
#Check and see if the message is too big if the maxsize for this account is set
if(maxsize != -1):
filesize = os.path.getsize(file)
if(filesize > maxsize):
size = os.path.getsize(os.path.join(self.getfullname(), file))
if(size > maxsize):
continue
foldermatch = messagename.find(folderstr) != -1
if not foldermatch:
@ -166,11 +170,13 @@ class MaildirFolder(BaseFolder):
nouidcounter -= 1
else:
uid = long(uidmatch.group(1))
#identify flags in the path name
flagmatch = self.flagmatchre.search(messagename)
flags = []
if flagmatch:
flags = [x for x in flagmatch.group(1)]
flags.sort()
flags = set(flagmatch.group(1))
else:
flags = set()
# 'filename' is 'dirannex/filename', e.g. cur/123_U=1_FMD5=1:2,S
retval[uid] = {'uid': uid,
'flags': flags,
'filename': file}
@ -261,7 +267,7 @@ class MaildirFolder(BaseFolder):
if rtime != None:
os.utime(os.path.join(tmpdir, messagename), (rtime, rtime))
self.messagelist[uid] = {'uid': uid, 'flags': [],
self.messagelist[uid] = {'uid': uid, 'flags': set(),
'filename': os.path.join('tmp', messagename)}
# savemessageflags moves msg to 'cur' or 'new' as appropriate
self.savemessageflags(uid, flags)
@ -288,14 +294,19 @@ class MaildirFolder(BaseFolder):
infostr = infomatch.group(1)
newname = newname.split(self.infosep)[0] # Strip off the info string.
infostr = re.sub('2,[A-Z]*', '', infostr)
flags.sort()
infostr += '2,' + ''.join(flags)
infostr += '2,' + ''.join(sorted(flags))
newname += infostr
newfilename = os.path.join(dir_prefix, newname)
if (newfilename != oldfilename):
os.rename(os.path.join(self.getfullname(), oldfilename),
os.path.join(self.getfullname(), newfilename))
try:
os.rename(os.path.join(self.getfullname(), oldfilename),
os.path.join(self.getfullname(), newfilename))
except OSError, e:
raise OfflineImapError("Can't rename file '%s' to '%s': %s" % (
oldfilename, newfilename, e[1]),
OfflineImapError.ERROR.FOLDER)
self.messagelist[uid]['flags'] = flags
self.messagelist[uid]['filename'] = newfilename

2
offlineimap/folder/UIDMaps.py
View File

@ -34,7 +34,7 @@ class MappedIMAPFolder(IMAPFolder):
diskl2r: dict mapping message uids: self.r2l[localuid]=remoteuid"""
def __init__(self, *args, **kwargs):
IMAPFolder.__init__(self, *args, **kwargs)
IMAPFolder.__init__(self, *args, **kwargs)
self.maplock = Lock()
(self.diskr2l, self.diskl2r) = self._loadmaps()
self._mb = IMAPFolder(*args, **kwargs)

241
offlineimap/imaplib2.py
View File

@ -2,7 +2,7 @@
"""Threaded IMAP4 client.
Based on RFC 2060 and original imaplib module.
Based on RFC 3501 and original imaplib module.
Public classes: IMAP4
IMAP4_SSL
@ -17,9 +17,9 @@ Public functions: Internaldate2Time
__all__ = ("IMAP4", "IMAP4_SSL", "IMAP4_stream",
"Internaldate2Time", "ParseFlags", "Time2Internaldate")
__version__ = "2.24"
__version__ = "2.28"
__release__ = "2"
__revision__ = "24"
__revision__ = "28"
__credits__ = """
Authentication code contributed by Donn Cave <donn@u.washington.edu> June 1998.
String method conversion by ESR, February 2001.
@ -38,7 +38,8 @@ Improved timeout handling contributed by Ivan Vovnenko <ivovnenko@gmail.com> Oct
Timeout handling further improved by Ethan Glasser-Camp <glasse@cs.rpi.edu> December 2010.
Time2Internaldate() patch to match RFC2060 specification of English month names from bugs.python.org/issue11024 March 2011.
starttls() bug fixed with the help of Sebastian Spaeth <sebastian@sspaeth.de> April 2011.
Threads now set the "daemon" flag (suggested by offlineimap-project)."""
Threads now set the "daemon" flag (suggested by offlineimap-project) April 2011.
Single quoting introduced with the help of Vladimir Marek <vladimir.marek@oracle.com> August 2011."""
__author__ = "Piers Lauder <piers@janeelix.com>"
__URL__ = "http://imaplib2.sourceforge.net"
__license__ = "Python License"
@ -57,7 +58,7 @@ IMAP4_SSL_PORT = 993
IDLE_TIMEOUT_RESPONSE = '* IDLE TIMEOUT\r\n'
IDLE_TIMEOUT = 60*29 # Don't stay in IDLE state longer
READ_POLL_TIMEOUT = 30 # Without this timeout interrupted network connections can hang reader
READ_SIZE = 32768 # Consume all available in socket
READ_SIZE = 32768 # Consume all available in socket
DFLT_DEBUG_BUF_LVL = 3 # Level above which the logging output goes directly to stderr
@ -88,7 +89,7 @@ Commands = {
'GETANNOTATION':((AUTH, SELECTED), True),
'GETQUOTA': ((AUTH, SELECTED), True),
'GETQUOTAROOT': ((AUTH, SELECTED), True),
'ID': ((NONAUTH, AUTH, SELECTED), True),
'ID': ((NONAUTH, AUTH, LOGOUT, SELECTED), True),
'IDLE': ((SELECTED,), False),
'LIST': ((AUTH, SELECTED), True),
'LOGIN': ((NONAUTH,), False),
@ -137,11 +138,14 @@ class Request(object):
"""Private class to represent a request awaiting response."""
def __init__(self, parent, name=None, callback=None, cb_arg=None):
def __init__(self, parent, name=None, callback=None, cb_arg=None, cb_self=False):
self.parent = parent
self.name = name
self.callback = callback # Function called to process result
self.callback_arg = cb_arg # Optional arg passed to "callback"
self.callback = callback # Function called to process result
if not cb_self:
self.callback_arg = cb_arg # Optional arg passed to "callback"
else:
self.callback_arg = (self, cb_arg) # Self reference required in callback arg
self.tag = '%s%s' % (parent.tagpre, parent.tagnum)
parent.tagnum += 1
@ -153,9 +157,6 @@ class Request(object):
def abort(self, typ, val):
"""Called whenever we abort a command
Sets self.aborted reason, and deliver()s nothing"""
self.aborted = (typ, val)
self.deliver(None)
@ -238,12 +239,17 @@ class IMAP4(object):
All (non-callback) arguments to commands are converted to strings,
except for AUTHENTICATE, and the last argument to APPEND which is
passed as an IMAP4 literal. If necessary (the string contains any
non-printing characters or white-space and isn't enclosed with either
parentheses or double quotes) each string is quoted. However, the
'password' argument to the LOGIN command is always quoted. If you
want to avoid having an argument string quoted (eg: the 'flags'
argument to STORE) then enclose the string in parentheses (eg:
"(\Deleted)").
non-printing characters or white-space and isn't enclosed with
either parentheses or double or single quotes) each string is
quoted. However, the 'password' argument to the LOGIN command is
always quoted. If you want to avoid having an argument string
quoted (eg: the 'flags' argument to STORE) then enclose the string
in parentheses (eg: "(\Deleted)"). If you are using "sequence sets"
containing the wildcard character '*', then enclose the argument
in single quotes: the quotes will be removed and the resulting
string passed unquoted. Note also that you can pass in an argument
with a type that doesn't evaluate to 'basestring' (eg: 'bytearray')
and it will be converted to a string without quoting.
There is one instance variable, 'state', that is useful for tracking
whether the client needs to login to the server. If it has the
@ -275,6 +281,7 @@ class IMAP4(object):
# so match not the inverse set
mustquote_cre = re.compile(r"[^!#$&'+,./0-9:;<=>?@A-Z\[^_`a-z|}~-]")
response_code_cre = re.compile(r'\[(?P<type>[A-Z-]+)( (?P<data>[^\]]*))?\]')
# sequence_set_cre = re.compile(r"^[0-9]+(:([0-9]+|\*))?(,[0-9]+(:([0-9]+|\*))?)*$")
untagged_response_cre = re.compile(r'\* (?P<type>[A-Z-]+)( (?P<data>.*))?')
untagged_status_cre = re.compile(r'\* (?P<data>\d+) (?P<type>[A-Z-]+)( (?P<data2>.*))?')
@ -339,8 +346,6 @@ class IMAP4(object):
self.state_change_free = threading.Event()
self.state_change_pending = threading.Lock()
self.commands_lock = threading.Lock()
"""commands_lock prevents self.untagged_responses to be
manipulated concurrently"""
self.idle_lock = threading.Lock()
self.ouq = Queue.Queue(10)
@ -368,7 +373,7 @@ class IMAP4(object):
elif self._get_untagged_response('OK'):
if __debug__: self._log(1, 'state => NONAUTH')
else:
raise self.error(self.welcome)
raise self.error('unrecognised server welcome message: %s' % `self.welcome`)
typ, dat = self.capability()
if dat == [None]:
@ -443,6 +448,35 @@ class IMAP4(object):
return s
def ssl_wrap_socket(self):
# Allow sending of keep-alive messages - seems to prevent some servers
# from closing SSL, leading to deadlocks.
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
try:
import ssl
if self.ca_certs is not None:
cert_reqs = ssl.CERT_REQUIRED
else:
cert_reqs = ssl.CERT_NONE
self.sock = ssl.wrap_socket(self.sock, self.keyfile, self.certfile, ca_certs=self.ca_certs, cert_reqs=cert_reqs)
ssl_exc = ssl.SSLError
except ImportError:
# No ssl module, and socket.ssl does not allow certificate verification
if self.ca_certs is not None:
raise socket.sslerror("SSL CA certificates cannot be checked without ssl module")
self.sock = socket.ssl(self.sock, self.keyfile, self.certfile)
ssl_exc = socket.sslerror
if self.cert_verify_cb is not None:
cert_err = self.cert_verify_cb(self.sock.getpeercert(), self.host)
if cert_err:
raise ssl_exc(cert_err)
self.read_fd = self.sock.fileno()
def start_compressing(self):
"""start_compressing()
Enable deflate compression on the socket (RFC 4978)."""
@ -671,7 +705,7 @@ class IMAP4(object):
def examine(self, mailbox='INBOX', **kw):
"""(typ, [data]) = examine(mailbox='INBOX', readonly=False)
"""(typ, [data]) = examine(mailbox='INBOX')
Select a mailbox for READ-ONLY access. (Flushes all untagged responses.)
'data' is count of messages in mailbox ('EXISTS' response).
Mandated responses are ('FLAGS', 'EXISTS', 'RECENT', 'UIDVALIDITY'), so
@ -745,13 +779,23 @@ class IMAP4(object):
def id(self, *kv_pairs, **kw):
"""(typ, [data]) = <instance>.id(kv_pairs)
'data' is list of ID key value pairs.
Request information for problem analysis and determination.
'kv_pairs' is a possibly empty list of keys and values.
'data' is a list of ID key value pairs or NIL.
NB: a single argument is assumed to be correctly formatted and is passed through unchanged
(for backward compatibility with earlier version).
Exchange information for problem analysis and determination.
The ID extension is defined in RFC 2971. """
name = 'ID'
kw['untagged_response'] = name
return self._simple_command(name, *kv_pairs, **kw)
if not kv_pairs:
data = 'NIL'
elif len(kv_pairs) == 1:
data = kv_pairs[0] # Assume invoker passing correctly formatted string (back-compat)
else:
data = '(%s)' % ' '.join([(arg and self._quote(arg) or 'NIL') for arg in kv_pairs])
return self._simple_command(name, (data,), **kw)
def idle(self, timeout=None, **kw):
@ -999,8 +1043,8 @@ class IMAP4(object):
return self._simple_command(name, sort_criteria, charset, *search_criteria, **kw)
def starttls(self, keyfile=None, certfile=None, **kw):
"""(typ, [data]) = starttls(keyfile=None, certfile=None)
def starttls(self, keyfile=None, certfile=None, ca_certs=None, cert_verify_cb=None, **kw):
"""(typ, [data]) = starttls(keyfile=None, certfile=None, ca_certs=None, cert_verify_cb=None)
Start TLS negotiation as per RFC 2595."""
name = 'STARTTLS'
@ -1013,7 +1057,7 @@ class IMAP4(object):
# Must now shutdown reader thread after next response, and restart after changing read_fd
self.read_size = 1 # Don't consume TLS handshake
self.read_size = 1 # Don't consume TLS handshake
self.TerminateReader = True
try:
@ -1031,14 +1075,13 @@ class IMAP4(object):
self.rdth.start()
raise self.error("Couldn't establish TLS session: %s" % dat)
try:
try:
import ssl
self.sock = ssl.wrap_socket(self.sock, keyfile, certfile)
except ImportError:
self.sock = socket.ssl(self.sock, keyfile, certfile)
self.keyfile = keyfile
self.certfile = certfile
self.ca_certs = ca_certs
self.cert_verify_cb = cert_verify_cb
self.read_fd = self.sock.fileno()
try:
self.ssl_wrap_socket()
finally:
# Restart reader thread
self.rdth = threading.Thread(target=self._reader)
@ -1140,29 +1183,34 @@ class IMAP4(object):
def _append_untagged(self, typ, dat):
"""Append new untagged response
Append new 'dat' to end of last untagged response if same 'typ',
else append new response."""
# Append new 'dat' to end of last untagged response if same 'typ',
# else append new response.
if dat is None: dat = ''
ur_data = []
self.commands_lock.acquire() # protect untagged_responses
self.commands_lock.acquire()
if self.untagged_responses and self.untagged_responses[-1][0] == typ:
# last respons is of type 'typ', get ur_data for appending
ur_data = self.untagged_responses[-1][1]
if self.untagged_responses:
urn, urd = self.untagged_responses[-1]
if urn != typ:
urd = None
else:
# need to create new untagged response of this type
self.untagged_responses.append([typ, ur_data])
urd = None
if urd is None:
urd = []
self.untagged_responses.append([typ, urd])
urd.append(dat)
ur_data.append(dat)
self.commands_lock.release()
if __debug__: self._log(5, 'untagged_responses[%s] %s += ["%s"]' % (typ, len(ur_data)-1, dat))
if __debug__: self._log(5, 'untagged_responses[%s] %s += ["%s"]' % (typ, len(urd)-1, dat))
def _check_bye(self):
"""raise Exception if untagged responses contains a 'BYE'"""
bye = self._get_untagged_response('BYE', leave=True)
if bye:
raise self.abort(bye[-1])
@ -1171,12 +1219,14 @@ class IMAP4(object):
def _checkquote(self, arg):
# Must quote command args if "atom-specials" present,
# and not already quoted.
# and not already quoted. NB: single quotes are removed.
if not isinstance(arg, basestring):
return arg
if len(arg) >= 2 and (arg[0],arg[-1]) in (('(',')'),('"','"')):
return arg
if len(arg) >= 2 and (arg[0],arg[-1]) in (("'","'"),):
return arg[1:-1]
if arg and self.mustquote_cre.search(arg) is None:
return arg
return self._quote(arg)
@ -1372,11 +1422,7 @@ class IMAP4(object):
def _get_untagged_response(self, name, leave=False):
"""Return an untagged response of type 'name'
:param leave: If leave (default: False) is True, we keep the
fetched responsem; otherwise it will be deleted. Returns
None if no such response found."""
self.commands_lock.acquire()
for i, (typ, dat) in enumerate(self.untagged_responses):
@ -1543,24 +1589,13 @@ class IMAP4(object):
def _simple_command(self, name, *args, **kw):
if 'callback' in kw:
rqb = self._command(name, callback=self._command_completer, *args)
rqb.callback_arg = (rqb, kw)
self._command(name, *args, callback=self._command_completer, cb_arg=kw, cb_self=True)
return (None, None)
return self._command_complete(self._command(name, *args), kw)
def _untagged_response(self, typ, dat, name):
"""Returns an untagged response for 'name' of type 'typ'
:param typ: 'OK, 'NO', etc... which will be used for the type of
the response.
:param dat: The fallback data to be used in case `typ` is
'NO'. Otherwise the data from the existing untagged
responses will be searched for data to be returned. If there
is no such response, we return `[None]` as data.
:param name: The name of the response.
:returns: (typ, data)
"""
if typ == 'NO':
return typ, dat
data = self._get_untagged_response(name)
@ -1936,22 +1971,27 @@ class IMAP4_SSL(IMAP4):
Instantiate with:
IMAP4_SSL(host=None, port=None, keyfile=None, certfile=None, debug=None, debug_file=None, identifier=None, timeout=None)
host - host's name (default: localhost);
port - port number (default: standard IMAP4 SSL port);
keyfile - PEM formatted file that contains your private key (default: None);
certfile - PEM formatted certificate chain file (default: None);
debug - debug level (default: 0 - no debug);
debug_file - debug stream (default: sys.stderr);
identifier - thread identifier prefix (default: host);
timeout - timeout in seconds when expecting a command response.
host - host's name (default: localhost);
port - port number (default: standard IMAP4 SSL port);
keyfile - PEM formatted file that contains your private key (default: None);
certfile - PEM formatted certificate chain file (default: None);
ca_certs - PEM formatted certificate chain file used to validate server certificates (default: None);
cert_verify_cb - function to verify authenticity of server certificates (default: None);
debug - debug level (default: 0 - no debug);
debug_file - debug stream (default: sys.stderr);
identifier - thread identifier prefix (default: host);
timeout - timeout in seconds when expecting a command response.
debug_buf_lvl - debug level at which buffering is turned off.
For more documentation see the docstring of the parent class IMAP4.
"""
def __init__(self, host=None, port=None, keyfile=None, certfile=None, debug=None, debug_file=None, identifier=None, timeout=None, debug_buf_lvl=None):
def __init__(self, host=None, port=None, keyfile=None, certfile=None, ca_certs=None, cert_verify_cb=None, debug=None, debug_file=None, identifier=None, timeout=None, debug_buf_lvl=None):
self.keyfile = keyfile
self.certfile = certfile
self.ca_certs = ca_certs
self.cert_verify_cb = cert_verify_cb
IMAP4.__init__(self, host, port, debug, debug_file, identifier, timeout, debug_buf_lvl)
@ -1965,14 +2005,7 @@ class IMAP4_SSL(IMAP4):
self.host = self._choose_nonull_or_dflt('', host)
self.port = self._choose_nonull_or_dflt(IMAP4_SSL_PORT, port)
self.sock = self.open_socket()
try:
import ssl
self.sslobj = ssl.wrap_socket(self.sock, self.keyfile, self.certfile)
except ImportError:
self.sslobj = socket.ssl(self.sock, self.keyfile, self.certfile)
self.read_fd = self.sock.fileno()
self.ssl_wrap_socket()
def read(self, size):
@ -1980,12 +2013,12 @@ class IMAP4_SSL(IMAP4):
Read at most 'size' bytes from remote."""
if self.decompressor is None:
return self.sslobj.read(size)
return self.sock.read(size)
if self.decompressor.unconsumed_tail:
data = self.decompressor.unconsumed_tail
else:
data = self.sslobj.read(8192)
data = self.sock.read(8192)
return self.decompressor.decompress(data, size)
@ -1998,21 +2031,23 @@ class IMAP4_SSL(IMAP4):
data = self.compressor.compress(data)
data += self.compressor.flush(zlib.Z_SYNC_FLUSH)
# NB: socket.ssl needs a "sendall" method to match socket objects.
bytes = len(data)
while bytes > 0:
sent = self.sslobj.write(data)
if sent == bytes:
break # avoid copy
data = data[sent:]
bytes = bytes - sent
if hasattr(self.sock, "sendall"):
self.sock.sendall(data)
else:
bytes = len(data)
while bytes > 0:
sent = self.sock.write(data)
if sent == bytes:
break # avoid copy
data = data[sent:]
bytes = bytes - sent
def ssl(self):
"""ssl = ssl()
Return socket.ssl instance used to communicate with the IMAP4 server."""
return self.sslobj
return self.sock
@ -2021,13 +2056,14 @@ class IMAP4_stream(IMAP4):
"""IMAP4 client class over a stream
Instantiate with:
IMAP4_stream(command, debug=None, debug_file=None, identifier=None, timeout=None)
IMAP4_stream(command, debug=None, debug_file=None, identifier=None, timeout=None, debug_buf_lvl=None)
command - string that can be passed to subprocess.Popen();
debug - debug level (default: 0 - no debug);
debug_file - debug stream (default: sys.stderr);
identifier - thread identifier prefix (default: host);
timeout - timeout in seconds when expecting a command response.
command - string that can be passed to subprocess.Popen();
debug - debug level (default: 0 - no debug);
debug_file - debug stream (default: sys.stderr);
identifier - thread identifier prefix (default: host);
timeout - timeout in seconds when expecting a command response.
debug_buf_lvl - debug level at which buffering is turned off.
For more documentation see the docstring of the parent class IMAP4.
"""
@ -2296,7 +2332,7 @@ if __name__ == '__main__':
('list', ('/tmp', 'imaplib2_test*')),
('select', ('/tmp/imaplib2_test.2',)),
('search', (None, 'SUBJECT', 'IMAP4 test')),
('fetch', ('1', '(FLAGS INTERNALDATE RFC822)')),
('fetch', ("'1:*'", '(FLAGS INTERNALDATE RFC822)')),
('store', ('1', 'FLAGS', '(\Deleted)')),
('namespace', ()),
('expunge', ()),
@ -2380,6 +2416,11 @@ if __name__ == '__main__':
else: path = ml.split()[-1]
run('delete', (path,))
if 'ID' in M.capabilities:
run('id', ())
run('id', ('("name", "imaplib2")',))
run('id', ("version", __version__, "os", os.uname()[0]))
for cmd,args in test_seq2:
if (cmd,args) != ('uid', ('SEARCH', 'SUBJECT', 'IMAP4 test')):
run(cmd, args)

182
offlineimap/imaplibutil.py
View File

@ -21,8 +21,10 @@ import re
import socket
import time
import subprocess
from offlineimap.ui import getglobalui
import threading
from hashlib import sha1
from offlineimap.ui import getglobalui
from offlineimap import OfflineImapError
from offlineimap.imaplib2 import IMAP4, IMAP4_SSL, zlib, IMAP4_PORT, InternalDate, Mon2num
@ -49,7 +51,14 @@ class UsefulIMAPMixIn:
return
# Wipe out all old responses, to maintain semantics with old imaplib2
del self.untagged_responses[:]
result = self.__class__.__bases__[1].select(self, mailbox, readonly)
try:
result = self.__class__.__bases__[1].select(self, mailbox, readonly)
except self.abort, e:
# self.abort is raised when we are supposed to retry
errstr = "Server '%s' closed connection, error on SELECT '%s'. Ser"\
"ver said: %s" % (self.host, mailbox, e.args[0])
severity = OfflineImapError.ERROR.FOLDER_RETRY
raise OfflineImapError(errstr, severity)
if result[0] != 'OK':
#in case of error, bail out with OfflineImapError
errstr = "Error SELECTing mailbox '%s', server reply:\n%s" %\
@ -127,164 +136,33 @@ def new_mesg(self, s, tn=None, secs=None):
tm = time.strftime('%M:%S', time.localtime(secs))
getglobalui().debug('imap', ' %s.%02d %s %s' % (tm, (secs*100)%100, tn, s))
class WrappedIMAP4_SSL(UsefulIMAPMixIn, IMAP4_SSL):
"""Provides an improved version of the standard IMAP4_SSL
It provides a better readline() implementation as impaplib's
readline() is extremly inefficient. It can also connect to IPv6
addresses."""
class WrappedIMAP4_SSL(UsefulIMAPMixIn, IMAP4_SSL):
"""Improved version of imaplib.IMAP4_SSL overriding select()"""
def __init__(self, *args, **kwargs):
self._readbuf = ''
self._cacertfile = kwargs.get('cacertfile', None)
if kwargs.has_key('cacertfile'):
del kwargs['cacertfile']
IMAP4_SSL.__init__(self, *args, **kwargs)
self._fingerprint = kwargs.get('fingerprint', None)
if kwargs.has_key('fingerprint'):
del kwargs['fingerprint']
super(WrappedIMAP4_SSL, self).__init__(*args, **kwargs)
def open(self, host=None, port=None):
"""Do whatever IMAP4_SSL would do in open, but call sslwrap
with cert verification"""
#IMAP4_SSL.open(self, host, port) uses the below 2 lines:
self.host = host
self.port = port
super(WrappedIMAP4_SSL, self).open(host, port)
if (self._fingerprint or not self.ca_certs) and\
'ssl' in locals(): # <--disable for python 2.5
# compare fingerprints
fingerprint = sha1(self.sock.getpeercert(True)).hexdigest()
if fingerprint != self._fingerprint:
raise OfflineImapError("Server SSL fingerprint '%s' for hostnam"
"e '%s' does not match configured fingerprint. Please ver"
"ify and set 'cert_fingerprint' accordingly if not set ye"
"t." % (fingerprint, host),
OfflineImapError.ERROR.REPO)
#rather than just self.sock = socket.create_connection((host, port))
#we use the below part to be able to connect to ipv6 addresses too
#This connects to the first ip found ipv4/ipv6
#Added by Adriaan Peeters <apeeters@lashout.net> based on a socket
#example from the python documentation:
#http://www.python.org/doc/lib/socket-example.html
res = socket.getaddrinfo(host, port, socket.AF_UNSPEC,
socket.SOCK_STREAM)
# Try all the addresses in turn until we connect()
last_error = 0
for remote in res:
af, socktype, proto, canonname, sa = remote
self.sock = socket.socket(af, socktype, proto)
last_error = self.sock.connect_ex(sa)
if last_error == 0:
break
else:
self.sock.close()
if last_error != 0:
raise Exception("can't open socket; error: %s"\
% socket.error(last_error))
# Allow sending of keep-alive message seems to prevent some servers
# from closing SSL on us leading to deadlocks
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
#connected to socket, now wrap it in SSL
try:
if self._cacertfile:
requirecert = ssl.CERT_REQUIRED
else:
requirecert = ssl.CERT_NONE
self.sslobj = ssl.wrap_socket(self.sock, self.keyfile,
self.certfile,
ca_certs = self._cacertfile,
cert_reqs = requirecert)
except NameError:
#Python 2.4/2.5 don't have the ssl module, we need to
#socket.ssl() here but that doesn't allow cert
#verification!!!
if self._cacertfile:
#user configured a CA certificate, but python 2.4/5 doesn't
#allow us to easily check it. So bail out here.
raise Exception("SSL CA Certificates cannot be checked with python <=2.6. Abort")
self.sslobj = socket.ssl(self.sock, self.keyfile,
self.certfile)
else:
#ssl.wrap_socket worked and cert is verified (if configured),
#now check that hostnames also match if we have a CA cert.
if self._cacertfile:
error = self._verifycert(self.sslobj.getpeercert(), host)
if error:
raise ssl.SSLError("SSL Certificate host name mismatch: %s" % error)
# imaplib2 uses this to poll()
self.read_fd = self.sock.fileno()
#TODO: Done for now. We should implement a mutt-like behavior
#that offers the users to accept a certificate (presenting a
#fingerprint of it) (get via self.sslobj.getpeercert()), and
#save that, and compare on future connects, rather than having
#to trust what the CA certs say.
def _verifycert(self, cert, hostname):
'''Verify that cert (in socket.getpeercert() format) matches hostname.
CRLs are not handled.
Returns error message if any problems are found and None on success.
'''
if not cert:
return ('no certificate received')
dnsname = hostname.lower()
certnames = []
# cert expired?
notafter = cert.get('notAfter')
if notafter:
if time.time() >= ssl.cert_time_to_seconds(notafter):
return ('server certificate error: certificate expired %s'
) % notafter
# First read commonName
for s in cert.get('subject', []):
key, value = s[0]
if key == 'commonName':
certnames.append(value.lower())
if len(certnames) == 0:
return ('no commonName found in certificate')
# Then read subjectAltName
for key, value in cert.get('subjectAltName', []):
if key == 'DNS':
certnames.append(value.lower())
# And finally try to match hostname with one of these names
for certname in certnames:
if (certname == dnsname or
'.' in dnsname and certname == '*.' + dnsname.split('.', 1)[1]):
return None
return ('no matching domain name found in certificate')
class WrappedIMAP4(UsefulIMAPMixIn, IMAP4):
"""Improved version of imaplib.IMAP4 that can also connect to IPv6"""
"""Improved version of imaplib.IMAP4 overriding select()"""
pass
def open(self, host = '', port = IMAP4_PORT):
"""Setup connection to remote server on "host:port"
(default: localhost:standard IMAP4 port).
"""
#self.host and self.port are needed by the parent IMAP4 class
self.host = host
self.port = port
res = socket.getaddrinfo(host, port, socket.AF_UNSPEC,
socket.SOCK_STREAM)
# Try each address returned by getaddrinfo in turn until we
# manage to connect to one.
# Try all the addresses in turn until we connect()
last_error = 0
for remote in res:
af, socktype, proto, canonname, sa = remote
self.sock = socket.socket(af, socktype, proto)
last_error = self.sock.connect_ex(sa)
if last_error == 0:
break
else:
self.sock.close()
if last_error != 0:
raise Exception("can't open socket; error: %s"\
% socket.error(last_error))
self.file = self.sock.makefile('rb')
# imaplib2 uses this to poll()
self.read_fd = self.sock.fileno()
mustquote = re.compile(r"[^\w!#$%&'+,.:;<=>?^`|~-]")
def Internaldate2epoch(resp):
"""Convert IMAP4 INTERNALDATE to UT.

275
offlineimap/imapserver.py
View File

@ -24,10 +24,12 @@ import offlineimap.accounts
import hmac
import socket
import base64
import time
import errno
from sys import exc_info
from socket import gaierror
try:
from ssl import SSLError
from ssl import SSLError, cert_time_to_seconds
except ImportError:
# Protect against python<2.6, use dummy and won't get SSL errors.
SSLError = None
@ -42,58 +44,58 @@ except ImportError:
pass
class IMAPServer:
"""Initializes all variables from an IMAPRepository() instance
Various functions, such as acquireconnection() return an IMAP4
object on which we can operate."""
GSS_STATE_STEP = 0
GSS_STATE_WRAP = 1
def __init__(self, config, reposname,
username = None, password = None, hostname = None,
port = None, ssl = 1, maxconnections = 1, tunnel = None,
reference = '""', sslclientcert = None, sslclientkey = None,
sslcacertfile = None, idlefolders = []):
def __init__(self, repos):
self.ui = getglobalui()
self.reposname = reposname
self.config = config
self.username = username
self.password = password
self.repos = repos
self.config = repos.getconfig()
self.tunnel = repos.getpreauthtunnel()
self.usessl = repos.getssl()
self.username = repos.getuser()
self.password = None
self.passworderror = None
self.goodpassword = None
self.hostname = hostname
self.tunnel = tunnel
self.port = port
self.usessl = ssl
self.sslclientcert = sslclientcert
self.sslclientkey = sslclientkey
self.sslcacertfile = sslcacertfile
self.hostname = repos.gethost()
self.port = repos.getport()
if self.port == None:
self.port = 993 if self.usessl else 143
self.sslclientcert = repos.getsslclientcert()
self.sslclientkey = repos.getsslclientkey()
self.sslcacertfile = repos.getsslcacertfile()
if self.sslcacertfile is None:
self.verifycert = None # disable cert verification
self.delim = None
self.root = None
if port == None:
if ssl:
self.port = 993
else:
self.port = 143
self.maxconnections = maxconnections
self.maxconnections = repos.getmaxconnections()
self.availableconnections = []
self.assignedconnections = []
self.lastowner = {}
self.semaphore = BoundedSemaphore(self.maxconnections)
self.connectionlock = Lock()
self.reference = reference
self.idlefolders = idlefolders
self.reference = repos.getreference()
self.idlefolders = repos.getidlefolders()
self.gss_step = self.GSS_STATE_STEP
self.gss_vc = None
self.gssapi = False
def getpassword(self):
if self.goodpassword != None:
"""Returns the server password or None"""
if self.goodpassword != None: # use cached good one first
return self.goodpassword
if self.password != None and self.passworderror == None:
return self.password
return self.password # non-failed preconfigured one
self.password = self.ui.getpass(self.reposname,
self.config,
self.passworderror)
# get 1) configured password first 2) fall back to asking via UI
self.password = self.repos.getpassword() or \
self.ui.getpass(self.repos.getname(), self.config,
self.passworderror)
self.passworderror = None
return self.password
def getdelim(self):
@ -107,12 +109,15 @@ class IMAPServer:
return self.root
def releaseconnection(self, connection):
"""Releases a connection, returning it to the pool."""
def releaseconnection(self, connection, drop_conn=False):
"""Releases a connection, returning it to the pool.
:param drop_conn: If True, the connection will be released and
not be reused. This can be used to indicate broken connections."""
self.connectionlock.acquire()
self.assignedconnections.remove(connection)
# Don't reuse broken connections
if connection.Terminate:
if connection.Terminate or drop_conn:
connection.logout()
else:
self.availableconnections.append(connection)
@ -204,17 +209,21 @@ class IMAPServer:
success = 1
elif self.usessl:
self.ui.connecting(self.hostname, self.port)
imapobj = imaplibutil.WrappedIMAP4_SSL(self.hostname, self.port,
self.sslclientkey, self.sslclientcert,
fingerprint = self.repos.get_ssl_fingerprint()
imapobj = imaplibutil.WrappedIMAP4_SSL(self.hostname,
self.port,
self.sslclientkey,
self.sslclientcert,
self.sslcacertfile,
self.verifycert,
timeout=socket.getdefaulttimeout(),
cacertfile = self.sslcacertfile)
fingerprint=fingerprint
)
else:
self.ui.connecting(self.hostname, self.port)
imapobj = imaplibutil.WrappedIMAP4(self.hostname, self.port,
timeout=socket.getdefaulttimeout())
imapobj.mustquote = imaplibutil.mustquote
if not self.tunnel:
try:
# Try GSSAPI and continue if it fails
@ -260,7 +269,6 @@ class IMAPServer:
except imapobj.error, val:
self.passworderror = str(val)
raise
#self.password = None
if self.delim == None:
listres = imapobj.list(self.reference, '""')[1]
@ -292,8 +300,6 @@ class IMAPServer:
error..."""
self.semaphore.release()
#Make sure that this can be retried the next time...
self.passworderror = None
if(self.connectionlock.locked()):
self.connectionlock.release()
@ -304,20 +310,20 @@ class IMAPServer:
reason = "Could not resolve name '%s' for repository "\
"'%s'. Make sure you have configured the ser"\
"ver name correctly and that you are online."%\
(self.hostname, self.reposname)
(self.hostname, self.repos)
raise OfflineImapError(reason, severity)
elif SSLError and isinstance(e, SSLError) and e.errno == 1:
# SSL unknown protocol error
# happens e.g. when connecting via SSL to a non-SSL service
if self.port != 443:
if self.port != 993:
reason = "Could not connect via SSL to host '%s' and non-s"\
"tandard ssl port %d configured. Make sure you connect"\
" to the correct port." % (self.hostname, self.port)
else:
reason = "Unknown SSL protocol connecting to host '%s' for"\
"repository '%s'. OpenSSL responded:\n%s"\
% (self.hostname, self.reposname, e)
% (self.hostname, self.repos, e)
raise OfflineImapError(reason, severity)
elif isinstance(e, socket.error) and e.args[0] == errno.ECONNREFUSED:
@ -333,7 +339,8 @@ class IMAPServer:
if str(e)[:24] == "can't open socket; error":
raise OfflineImapError("Could not connect to remote server '%s' "\
"for repository '%s'. Remote does not answer."
% (self.hostname, self.reposname), severity)
% (self.hostname, self.repos),
OfflineImapError.ERROR.REPO)
else:
# re-raise all other errors
raise
@ -408,11 +415,56 @@ class IMAPServer:
self.ui.debug('imap', 'keepalive: bottom of loop')
def verifycert(self, cert, hostname):
'''Verify that cert (in socket.getpeercert() format) matches hostname.
CRLs are not handled.
Returns error message if any problems are found and None on success.
'''
errstr = "CA Cert verifying failed: "
if not cert:
return ('%s no certificate received' % errstr)
dnsname = hostname.lower()
certnames = []
# cert expired?
notafter = cert.get('notAfter')
if notafter:
if time.time() >= cert_time_to_seconds(notafter):
return '%s certificate expired %s' % (errstr, notafter)
# First read commonName
for s in cert.get('subject', []):
key, value = s[0]
if key == 'commonName':
certnames.append(value.lower())
if len(certnames) == 0:
return ('%s no commonName found in certificate' % errstr)
# Then read subjectAltName
for key, value in cert.get('subjectAltName', []):
if key == 'DNS':
certnames.append(value.lower())
# And finally try to match hostname with one of these names
for certname in certnames:
if (certname == dnsname or
'.' in dnsname and certname == '*.' + dnsname.split('.', 1)[1]):
return None
return ('%s no matching domain name found in certificate' % errstr)
class IdleThread(object):
def __init__(self, parent, folder=None):
"""If invoked without 'folder', perform a NOOP and wait for
self.stop() to be called. If invoked with folder, switch to IDLE
mode and synchronize once we have a new message"""
self.parent = parent
self.folder = folder
self.event = Event()
self.stop_sig = Event()
self.ui = getglobalui()
if folder is None:
self.thread = Thread(target=self.noop)
else:
@ -423,7 +475,7 @@ class IdleThread(object):
self.thread.start()
def stop(self):
self.event.set()
self.stop_sig.set()
def join(self):
self.thread.join()
@ -431,7 +483,7 @@ class IdleThread(object):
def noop(self):
imapobj = self.parent.acquireconnection()
imapobj.noop()
self.event.wait()
self.stop_sig.wait()
self.parent.releaseconnection(imapobj)
def dosync(self):
@ -446,87 +498,56 @@ class IdleThread(object):
ui.unregisterthread(currentThread())
def idle(self):
while True:
if self.event.isSet():
return
self.needsync = False
self.imapaborted = False
def callback(args):
result, cb_arg, exc_data = args
if exc_data is None:
if not self.event.isSet():
self.needsync = True
self.event.set()
else:
# We got an "abort" signal.
self.imapaborted = True
self.stop()
"""Invoke IDLE mode until timeout or self.stop() is invoked"""
def callback(args):
"""IDLE callback function invoked by imaplib2
imapobj = self.parent.acquireconnection()
imapobj.select(self.folder)
This is invoked when a) The IMAP server tells us something
while in IDLE mode, b) we get an Exception (e.g. on dropped
connections, or c) the standard imaplib IDLE timeout of 29
minutes kicks in."""
result, cb_arg, exc_data = args
if exc_data is None and not self.stop_sig.isSet():
# No Exception, and we are not supposed to stop:
self.needsync = True
self.stop_sig.set() # continue to sync
while not self.stop_sig.isSet():
self.needsync = False
success = False # successfully selected FOLDER?
while not success:
imapobj = self.parent.acquireconnection()
try:
imapobj.select(self.folder)
except OfflineImapError, e:
if e.severity == OfflineImapError.ERROR.FOLDER_RETRY:
# Connection closed, release connection and retry
self.ui.error(e, exc_info()[2])
self.parent.releaseconnection(imapobj, True)
else:
raise e
else:
success = True
if "IDLE" in imapobj.capabilities:
imapobj.idle(callback=callback)
else:
ui = getglobalui()
ui.warn("IMAP IDLE not supported on connection to %s."
"Falling back to old behavior: sleeping until next"
"refresh cycle."
%(imapobj.identifier,))
self.ui.warn("IMAP IDLE not supported on server '%s'."
"Sleep until next refresh cycle." % imapobj.identifier)
imapobj.noop()
self.event.wait()
if self.event.isSet():
# Can't NOOP on a bad connection.
if not self.imapaborted:
imapobj.noop()
# We don't do event.clear() so that we'll fall out
# of the loop next time around.
self.parent.releaseconnection(imapobj)
self.stop_sig.wait() # self.stop() or IDLE callback are invoked
try:
# End IDLE mode with noop, imapobj can point to a dropped conn.
imapobj.noop()
except imapobj.abort():
self.ui.warn('Attempting NOOP on dropped connection %s' % \
imapobj.identifier)
self.parent.releaseconnection(imapobj, True)
else:
self.parent.releaseconnection(imapobj)
if self.needsync:
self.event.clear()
# here not via self.stop, but because IDLE responded. Do
# another round and invoke actual syncing.
self.stop_sig.clear()
self.dosync()
class ConfigedIMAPServer(IMAPServer):
"""This class is designed for easier initialization given a ConfigParser
object and an account name. The passwordhash is used if
passwords for certain accounts are known. If the password for this
account is listed, it will be obtained from there."""
def __init__(self, repository, passwordhash = {}):
"""Initialize the object. If the account is not a tunnel,
the password is required."""
self.repos = repository
self.config = self.repos.getconfig()
usetunnel = self.repos.getpreauthtunnel()
if not usetunnel:
host = self.repos.gethost()
user = self.repos.getuser()
port = self.repos.getport()
ssl = self.repos.getssl()
sslclientcert = self.repos.getsslclientcert()
sslclientkey = self.repos.getsslclientkey()
sslcacertfile = self.repos.getsslcacertfile()
reference = self.repos.getreference()
idlefolders = self.repos.getidlefolders()
server = None
password = None
if repository.getname() in passwordhash:
password = passwordhash[repository.getname()]
# Connect to the remote server.
if usetunnel:
IMAPServer.__init__(self, self.config, self.repos.getname(),
tunnel = usetunnel,
reference = reference,
idlefolders = idlefolders,
maxconnections = self.repos.getmaxconnections())
else:
if not password:
password = self.repos.getpassword()
IMAPServer.__init__(self, self.config, self.repos.getname(),
user, password, host, port, ssl,
self.repos.getmaxconnections(),
reference = reference,
idlefolders = idlefolders,
sslclientcert = sslclientcert,
sslclientkey = sslclientkey,
sslcacertfile = sslcacertfile)

84
offlineimap/imaputil.py
View File

@ -20,6 +20,11 @@ import re
import string
import types
from offlineimap.ui import getglobalui
try: # python 2.6 has set() built in
set
except NameError:
from sets import Set as set
quotere = re.compile('^("(?:[^"]|\\\\")*")')
def debug(*args):
@ -42,11 +47,21 @@ def dequote(string):
return string
def flagsplit(string):
"""Converts a string of IMAP flags to a list
:returns: E.g. '(\\Draft \\Deleted)' returns ['\\Draft','\\Deleted'].
(FLAGS (\\Seen Old) UID 4807) returns
['FLAGS,'(\\Seen Old)','UID', '4807']
"""
if string[0] != '(' or string[-1] != ')':
raise ValueError, "Passed string '%s' is not a flag list" % string
return imapsplit(string[1:-1])
def options2hash(list):
"""convert list [1,2,3,4,5,6] to {1:2, 3:4, 5:6}"""
# effectively this does dict(zip(l[::2],l[1::2])), however
# measurements seemed to have indicated that the manual variant is
# faster for mosly small lists.
retval = {}
counter = 0
while (counter < len(list)):
@ -55,8 +70,12 @@ def options2hash(list):
debug("options2hash returning:", retval)
return retval
def flags2hash(string):
return options2hash(flagsplit(string))
def flags2hash(flags):
"""Converts IMAP response string from eg IMAP4.fetch() to a hash.
E.g. '(FLAGS (\\Seen Old) UID 4807)' leads to
{'FLAGS': '(\\Seen Old)', 'UID': '4807'}"""
return options2hash(flagsplit(flags))
def imapsplit(imapstring):
"""Takes a string from an IMAP conversation and returns a list containing
@ -152,15 +171,16 @@ flagmap = [('\\Seen', 'S'),
('\\Draft', 'D')]
def flagsimap2maildir(flagstring):
retval = []
imapflaglist = [x.lower() for x in flagstring[1:-1].split()]
"""Convert string '(\\Draft \\Deleted)' into a flags set(DR)"""
retval = set()
imapflaglist = flagstring[1:-1].split()
for imapflag, maildirflag in flagmap:
if imapflag.lower() in imapflaglist:
retval.append(maildirflag)
retval.sort()
if imapflag in imapflaglist:
retval.add(maildirflag)
return retval
def flagsmaildir2imap(maildirflaglist):
"""Convert set of flags ([DR]) into a string '(\\Draft \\Deleted)'"""
retval = []
for imapflag, maildirflag in flagmap:
if maildirflag in maildirflaglist:
@ -168,38 +188,32 @@ def flagsmaildir2imap(maildirflaglist):
retval.sort()
return '(' + ' '.join(retval) + ')'
def listjoin(list):
start = None
end = None
retval = []
def uid_sequence(uidlist):
"""Collapse UID lists into shorter sequence sets
def getlist(start, end):
[1,2,3,4,5,10,12,13] will return "1:5,10,12:13". This function sorts
the list, and only collapses if subsequent entries form a range.
:returns: The collapsed UID list as string"""
def getrange(start, end):
if start == end:
return(str(start))
else:
return(str(start) + ":" + str(end))
return "%s:%s" % (start, end)
for item in list:
if start == None:
# First item.
start = item
end = item
elif item == end + 1:
# An addition to the list.
end = item
else:
# Here on: starting a new list.
retval.append(getlist(start, end))
start = item
end = item
if not len(uidlist): return '' # Empty list, return
start, end = None, None
retval = []
# Force items to be longs and sort them
sorted_uids = sorted(map(int, uidlist))
if start != None:
retval.append(getlist(start, end))
for item in iter(sorted_uids):
item = int(item)
if start == None: # First item
start, end = item, item
elif item == end + 1: # Next item in a range
end = item
else: # Starting a new range
retval.append(getrange(start, end))
start, end = item, item
retval.append(getrange(start, end)) # Add final range/item
return ",".join(retval)

51
offlineimap/repository/Base.py
View File

@ -42,14 +42,14 @@ class BaseRepository(object, CustomConfig.ConfigHelperMixin):
# The 'restoreatime' config parameter only applies to local Maildir
# mailboxes.
def restore_atime(self):
if self.config.get('Repository ' + self.name, 'type').strip() != \
'Maildir':
return
if self.config.get('Repository ' + self.name, 'type').strip() != \
'Maildir':
return
if not self.config.has_option('Repository ' + self.name, 'restoreatime') or not self.config.getboolean('Repository ' + self.name, 'restoreatime'):
return
if not self.config.has_option('Repository ' + self.name, 'restoreatime') or not self.config.getboolean('Repository ' + self.name, 'restoreatime'):
return
return self.restore_folder_atimes()
return self.restore_folder_atimes()
def connect(self):
"""Establish a connection to the remote, if necessary. This exists
@ -114,37 +114,32 @@ class BaseRepository(object, CustomConfig.ConfigHelperMixin):
def getfolder(self, foldername):
raise NotImplementedError
def syncfoldersto(self, dest, copyfolders):
def syncfoldersto(self, dst_repo, status_repo):
"""Syncs the folders in this repository to those in dest.
It does NOT sync the contents of those folders.
For every time dest.makefolder() is called, also call makefolder()
on each folder in copyfolders."""
src = self
srcfolders = src.getfolders()
destfolders = dest.getfolders()
It does NOT sync the contents of those folders."""
src_repo = self
src_folders = src_repo.getfolders()
dst_folders = dst_repo.getfolders()
# Create hashes with the names, but convert the source folders
# to the dest folder's sep.
srchash = {}
for folder in srcfolders:
srchash[folder.getvisiblename().replace(src.getsep(), dest.getsep())] = \
folder
desthash = {}
for folder in destfolders:
desthash[folder.getvisiblename()] = folder
src_hash = {}
for folder in src_folders:
src_hash[folder.getvisiblename().replace(
src_repo.getsep(), dst_repo.getsep())] = folder
dst_hash = {}
for folder in dst_folders:
dst_hash[folder.getvisiblename()] = folder
#
# Find new folders.
#
for key in srchash.keys():
if not key in desthash:
for key in src_hash.keys():
if not key in dst_hash:
try:
dest.makefolder(key)
for copyfolder in copyfolders:
copyfolder.makefolder(key.replace(dest.getsep(), copyfolder.getsep()))
dst_repo.makefolder(key)
status_repo.makefolder(key.replace(dst_repo.getsep(),
status_repo.getsep()))
except (KeyboardInterrupt):
raise
except:

2
offlineimap/repository/Gmail.py
View File

@ -73,7 +73,7 @@ class GmailRepository(IMAPRepository):
def gettrashfolder(self, foldername):
#: Where deleted mail should be moved
return self.getconf('trashfolder','[Gmail]/Trash')
def getspamfolder(self):
#: Gmail also deletes messages upon EXPUNGE in the Spam folder
return self.getconf('spamfolder','[Gmail]/Spam')

13
offlineimap/repository/IMAP.py
View File

@ -24,6 +24,7 @@ from threading import Event
import re
import types
import os
from sys import exc_info
import netrc
import errno
@ -33,7 +34,7 @@ class IMAPRepository(BaseRepository):
BaseRepository.__init__(self, reposname, account)
# self.ui is being set by the BaseRepository
self._host = None
self.imapserver = imapserver.ConfigedIMAPServer(self)
self.imapserver = imapserver.IMAPServer(self)
self.folders = None
self.nametrans = lambda foldername: foldername
self.folderfilter = lambda foldername: 1
@ -181,6 +182,9 @@ class IMAPRepository(BaseRepository):
% (self.name, cacertfile))
return cacertfile
def get_ssl_fingerprint(self):
return self.getconf('cert_fingerprint', None)
def getpreauthtunnel(self):
return self.getconf('preauthtunnel', None)
@ -307,7 +311,12 @@ class IMAPRepository(BaseRepository):
for foldername in self.folderincludes:
try:
imapobj.select(foldername, readonly = 1)
except ValueError:
except OfflineImapError, e:
# couldn't select this folderinclude, so ignore folder.
if e.severity > OfflineImapError.ERROR.FOLDER:
raise
self.ui.error(e, exc_info()[2],
'Invalid folderinclude:')
continue
retval.append(self.getfoldertype()(self.imapserver,
foldername,

22
offlineimap/repository/LocalStatus.py
View File

@ -50,10 +50,17 @@ class LocalStatusRepository(BaseRepository):
return '.'
def getfolderfilename(self, foldername):
"""Return the full path of the status file"""
# replace with 'dot' if final path name is '.'
foldername = re.sub('(^|\/)\.$','\\1dot', foldername)
return os.path.join(self.directory, foldername)
"""Return the full path of the status file
This mimics the path that Folder().getfolderbasename() would return"""
if not foldername:
basename = '.'
else: #avoid directory hierarchies and file names such as '/'
basename = foldername.replace('/', '.')
# replace with literal 'dot' if final path name is '.' as '.' is
# an invalid file name.
basename = re.sub('(^|\/)\.$','\\1dot', basename)
return os.path.join(self.directory, basename)
def makefolder(self, foldername):
"""Create a LocalStatus Folder
@ -80,14 +87,13 @@ class LocalStatusRepository(BaseRepository):
self.config)
def getfolders(self):
"""Returns a list of ALL folders on this server.
This is currently nowhere used in the code."""
"""Returns a list of all cached folders."""
if self._folders != None:
return self._folders
self._folders = []
for folder in os.listdir(self.directory):
self._folders = retval.append(self.getfolder(folder))
self._folders.append(self.getfolder(folder))
return self._folders
def forgetfolders(self):

34
offlineimap/repository/Maildir.py
View File

@ -32,28 +32,28 @@ class MaildirRepository(BaseRepository):
self.folders = None
self.ui = getglobalui()
self.debug("MaildirRepository initialized, sep is " + repr(self.getsep()))
self.folder_atimes = []
self.folder_atimes = []
# Create the top-level folder if it doesn't exist
if not os.path.isdir(self.root):
os.mkdir(self.root, 0700)
def _append_folder_atimes(self, foldername):
p = os.path.join(self.root, foldername)
new = os.path.join(p, 'new')
cur = os.path.join(p, 'cur')
f = p, os.stat(new)[ST_ATIME], os.stat(cur)[ST_ATIME]
self.folder_atimes.append(f)
p = os.path.join(self.root, foldername)
new = os.path.join(p, 'new')
cur = os.path.join(p, 'cur')
f = p, os.stat(new)[ST_ATIME], os.stat(cur)[ST_ATIME]
self.folder_atimes.append(f)
def restore_folder_atimes(self):
if not self.folder_atimes:
return
if not self.folder_atimes:
return
for f in self.folder_atimes:
t = f[1], os.stat(os.path.join(f[0], 'new'))[ST_MTIME]
os.utime(os.path.join(f[0], 'new'), t)
t = f[2], os.stat(os.path.join(f[0], 'cur'))[ST_MTIME]
os.utime(os.path.join(f[0], 'cur'), t)
for f in self.folder_atimes:
t = f[1], os.stat(os.path.join(f[0], 'new'))[ST_MTIME]
os.utime(os.path.join(f[0], 'new'), t)
t = f[2], os.stat(os.path.join(f[0], 'cur'))[ST_MTIME]
os.utime(os.path.join(f[0], 'cur'), t)
def getlocalroot(self):
return os.path.expanduser(self.getconf('localfolders'))
@ -110,8 +110,8 @@ class MaildirRepository(BaseRepository):
self.ui.warn("NOT YET IMPLEMENTED: DELETE FOLDER %s" % foldername)
def getfolder(self, foldername):
if self.config.has_option('Repository ' + self.name, 'restoreatime') and self.config.getboolean('Repository ' + self.name, 'restoreatime'):
self._append_folder_atimes(foldername)
if self.config.has_option('Repository ' + self.name, 'restoreatime') and self.config.getboolean('Repository ' + self.name, 'restoreatime'):
self._append_folder_atimes(foldername)
return folder.Maildir.MaildirFolder(self.root, foldername,
self.getsep(), self,
self.accountname, self.config)
@ -155,11 +155,11 @@ class MaildirRepository(BaseRepository):
# This directory has maildir stuff -- process
self.debug(" This is maildir folder '%s'." % foldername)
if self.config.has_option('Repository %s' % self,
if self.config.has_option('Repository %s' % self,
'restoreatime') and \
self.config.getboolean('Repository %s' % self,
'restoreatime'):
self._append_folder_atimes(foldername)
self._append_folder_atimes(foldername)
retval.append(folder.Maildir.MaildirFolder(self.root,
foldername,
self.getsep(),

4
offlineimap/ui/Blinkenlights.py
View File

@ -54,9 +54,9 @@ class BlinkenBase:
s.gettf().setcolor('blue')
s.__class__.__bases__[-1].syncingmessages(s, sr, sf, dr, df)
def copyingmessage(s, uid, src, destlist):
def copyingmessage(s, uid, src, destfolder):
s.gettf().setcolor('orange')
s.__class__.__bases__[-1].copyingmessage(s, uid, src, destlist)
s.__class__.__bases__[-1].copyingmessage(s, uid, src, destfolder)
def deletingmessages(s, uidlist, destlist):
s.gettf().setcolor('red')

2
offlineimap/ui/Curses.py
View File

@ -230,7 +230,7 @@ class CursesThreadFrame:
if self.getcolor() == 'black':
self.window.addstr(self.y, self.x, ' ', self.color)
else:
self.window.addstr(self.y, self.x, self.ui.config.getdefault("ui.Curses.Blinkenlights", "statuschar", '.'), self.color)
self.window.addstr(self.y, self.x, '.', self.color)
self.c.stdscr.move(self.c.height - 1, self.c.width - 1)
self.window.refresh()
self.c.locked(lockedstuff)

8
offlineimap/ui/Machine.py
View File

@ -108,10 +108,10 @@ class MachineUI(UIBase):
(s.getnicename(sr), sf.getname(), s.getnicename(dr),
df.getname()))
def copyingmessage(s, uid, src, destlist):
ds = s.folderlist(destlist)
s._printData('copyingmessage', "%d\n%s\n%s\n%s" % \
(uid, s.getnicename(src), src.getname(), ds))
def copyingmessage(self, uid, srcfolder, destfolder):
self._printData('copyingmessage', "%d\n%s\n%s\n%s[%s]" % \
(uid, self.getnicename(srcfolder), srcfolder.getname(),
self.getnicename(destfolder), destfolder))
def folderlist(s, list):
return ("\f".join(["%s\t%s" % (s.getnicename(x), x.getname()) for x in list]))

71
offlineimap/ui/UIBase.py
View File

@ -21,6 +21,7 @@ import time
import sys
import traceback
import threading
from Queue import Queue
import offlineimap
debugtypes = {'':'Other offlineimap related sync messages',
@ -47,7 +48,9 @@ class UIBase:
s.debugmsglen = 50
s.threadaccounts = {}
s.logfile = None
s.exc_queue = Queue()
"""saves all occuring exceptions, so we can output them at the end"""
################################################## UTILS
def _msg(s, msg):
"""Generic tool called when no other works."""
@ -82,6 +85,39 @@ class UIBase:
else:
s._msg("WARNING: " + msg)
def error(self, exc, exc_traceback=None, msg=None):
"""Log a message at severity level ERROR
Log Exception 'exc' to error log, possibly prepended by a preceding
error "msg", detailing at what point the error occurred.
In debug mode, we also output the full traceback that occurred
if one has been passed in via sys.info()[2].
Also save the Exception to a stack that can be output at the end
of the sync run when offlineiamp exits. It is recommended to
always pass in exceptions if possible, so we can give the user
the best debugging info.
One example of such a call might be:
ui.error(exc, sys.exc_info()[2], msg="While syncing Folder %s in "
"repo %s")
"""
cur_thread = threading.currentThread()
if msg:
self._msg("ERROR [%s]: %s\n %s" % (cur_thread, msg, exc))
else:
self._msg("ERROR [%s]: %s" % (cur_thread, exc))
if not self.debuglist:
# only output tracebacks in debug mode
exc_traceback = None
# push exc on the queue for later output
self.exc_queue.put((msg, exc, exc_traceback))
if exc_traceback:
self._msg(traceback.format_tb(exc_traceback))
def registerthread(s, account):
"""Provides a hint to UIs about which account this particular
thread is processing."""
@ -249,11 +285,12 @@ class UIBase:
s.getnicename(dr),
df.getname()))
def copyingmessage(s, uid, src, destlist):
if s.verbose >= 0:
ds = s.folderlist(destlist)
s._msg("Copy message %d %s[%s] -> %s" % (uid, s.getnicename(src),
src.getname(), ds))
def copyingmessage(self, uid, src, destfolder):
"""Output a log line stating which message we copy"""
if self.verbose >= 0:
self._msg("Copy message %d %s[%s] -> %s[%s]" % \
(uid, self.getnicename(src), src,
self.getnicename(destfolder), destfolder))
def deletingmessage(s, uid, destlist):
if s.verbose >= 0:
@ -265,7 +302,7 @@ class UIBase:
ds = s.folderlist(destlist)
s._msg("Deleting %d messages (%s) in %s" % \
(len(uidlist),
", ".join([str(u) for u in uidlist]),
offlineimap.imaputil.uid_sequence(uidlist),
ds))
def addingflags(s, uidlist, flags, dest):
@ -315,12 +352,24 @@ class UIBase:
def mainException(s):
s._msg(s.getMainExceptionString())
def terminate(s, exitstatus = 0, errortitle = None, errormsg = None):
def terminate(self, exitstatus = 0, errortitle = None, errormsg = None):
"""Called to terminate the application."""
if errormsg <> None:
if errortitle <> None:
sys.stderr.write('ERROR: %s\n\n%s\n'%(errortitle, errormsg))
#print any exceptions that have occurred over the run
if not self.exc_queue.empty():
self._msg("\nERROR: Exceptions occurred during the run!")
while not self.exc_queue.empty():
msg, exc, exc_traceback = self.exc_queue.get()
if msg:
self._msg("ERROR: %s\n %s" % (msg, exc))
else:
self._msg("ERROR: %s" % (exc))
if exc_traceback:
self._msg("\nTraceback:\n%s" %"".join(
traceback.format_tb(exc_traceback)))
if errormsg and errortitle:
sys.stderr.write('ERROR: %s\n\n%s\n'%(errortitle, errormsg))
elif errormsg:
sys.stderr.write('%s\n' % errormsg)
sys.exit(exitstatus)