Merge branch 'next'

This commit is contained in:
Nicolas Sebrecht 2011-09-12 20:13:08 +02:00
commit e5a26dcfd8
31 changed files with 1211 additions and 930 deletions

View File

@ -19,8 +19,6 @@ Changes
Bug Fixes Bug Fixes
--------- ---------
Pending for the next major release Pending for the next major release
================================== ==================================

View File

@ -12,6 +12,45 @@ ChangeLog
releases announces. releases announces.
OfflineIMAP v6.3.5-rc1 (2011-09-12)
===================================
Notes
-----
Idle feature and SQLite backend leave the experimental stage! ,-)
New Features
------------
* When a message upload/download fails, we do not abort the whole folder
synchronization, but only skip that message, informing the user at the
end of the sync run.
* If you connect via ssl and 'cert_fingerprint' is configured, we check
that the server certificate is actually known and identical by
comparing the stored sha1 fingerprint with the current one.
Changes
-------
* Refactor our IMAPServer class. Background work without user-visible
changes.
* Remove the configurability of the Blinkenlights statuschar. It
cluttered the main configuration file for little gain.
* Updated bundled imaplib2 to version 2.28.
Bug Fixes
---------
* We protect more robustly against asking for inexistent messages from the
IMAP server, when someone else deletes or moves messages while we sync.
* Selecting inexistent folders specified in folderincludes now throws
nice errors and continues to sync with all other folders rather than
exiting offlineimap with a traceback.
OfflineIMAP v6.3.4 (2011-08-10) OfflineIMAP v6.3.4 (2011-08-10)
=============================== ===============================
@ -26,6 +65,7 @@ Changes
* Handle when UID can't be found on saved messages. * Handle when UID can't be found on saved messages.
OfflineIMAP v6.3.4-rc4 (2011-07-27) OfflineIMAP v6.3.4-rc4 (2011-07-27)
=================================== ===================================

View File

@ -14,39 +14,44 @@ Powerful IMAP/Maildir synchronization and reader support
.. TODO: :Manual group: .. TODO: :Manual group:
SYNOPSIS
========
offlineimap [-h|--help]
offlineimap [OPTIONS]
| -1
| -P profiledir
| -a accountlist
| -c configfile
| -d debugtype[,...]
| -f foldername[,...]
| -k [section:]option=value
| -l filename
| -o
| -u interface
DESCRIPTION DESCRIPTION
=========== ===========
Most configuration is done via the configuration file. Nevertheless, there are OfflineImap operates on a REMOTE and a LOCAL repository and synchronizes
a few command-line options that you may set for OfflineIMAP. emails between them, so that you can read the same mailbox from multiple
computers. The REMOTE repository is some IMAP server, while LOCAL can be
either a local Maildir or another IMAP server.
Missing folders will be automatically created on the LOCAL side, however
NO folders will currently be created on the REMOTE repository
automatically (it will sync your emails from local folders if
corresponding REMOTE folders already exist).
Configuring OfflineImap in basic mode is quite easy, however it provides
an amazing amount of flexibility for those with special needs. You can
specify the number of connections to your IMAP server, use arbitrary
python functions (including regular expressions) to limit the number of
folders being synchronized. You can transpose folder names between
repositories using any python function, to mangle and modify folder
names on the LOCAL repository. There are six different ways to hand the
IMAP password to OfflineImap from console input, specifying in the
configuration file, .netrc support, specifying in a separate file, to
using arbitrary python functions that somehow return the
password. Finally, you can use IMAPs IDLE infrastructure to always keep
a connection to your IMAP server open and immediately be notified (and
synchronized) when a new mail arrives (aka Push mail).
Most configuration is done via the configuration file. However, any setting can also be overriden by command line options handed to OfflineIMAP.
OfflineImap is well suited to be frequently invoked by cron jobs, or can run in daemon mode to periodically check your email (however, it will exit in some error situations).
Check out the `Use Cases`_ section for some example configurations.
OPTIONS OPTIONS
======= =======
-1 Disable most multithreading operations -1 Disable most multithreading operations
Use solely a single-connection sync. This effectively sets the Use solely a single-connection sync. This effectively sets the
@ -152,8 +157,7 @@ Blinkenlights
--------------- ---------------
Blinkenlights is an interface designed to be sleek, fun to watch, and Blinkenlights is an interface designed to be sleek, fun to watch, and
informative of the overall picture of what OfflineIMAP is doing. I consider it informative of the overall picture of what OfflineIMAP is doing.
to be the best general-purpose interface in OfflineIMAP.
Blinkenlights contains a row of "LEDs" with command buttons and a log. Blinkenlights contains a row of "LEDs" with command buttons and a log.
@ -230,30 +234,27 @@ English-speaking world. One version ran in its entirety as follows:
TTYUI TTYUI
--------- ---------
TTYUI interface is for people running in basic, non-color terminals. It TTYUI interface is for people running in terminals. It prints out basic
prints out basic status messages and is generally friendly to use on a console status messages and is generally friendly to use on a console or xterm.
or xterm.
Basic Basic
-------------------- ------
Basic is designed for situations in which OfflineIMAP will be run Basic is designed for situations in which OfflineIMAP will be run
non-attended and the status of its execution will be logged. You might use it, non-attended and the status of its execution will be logged. This user
for instance, to have the system run automatically and e-mail you the results of interface is not capable of reading a password from the keyboard;
the synchronization. This user interface is not capable of reading a password account passwords must be specified using one of the configuration file
from the keyboard; account passwords must be specified using one of the options.
configuration file options.
Quiet Quiet
----- -----
Quiet is designed for non-attended running in situations where normal It will output nothing except errors and serious warnings. Like Basic,
status messages are not desired. It will output nothing except errors this user interface is not capable of reading a password from the
and serious warnings. Like Basic, this user interface is not capable keyboard; account passwords must be specified using one of the
of reading a password from the keyboard; account passwords must be configuration file options.
specified using one of the configuration file options.
MachineUI MachineUI
--------- ---------
@ -262,8 +263,98 @@ MachineUI generates output in a machine-parsable format. It is designed
for other programs that will interface to OfflineIMAP. for other programs that will interface to OfflineIMAP.
Signals Synchronization Performance
======= ===========================
By default, we use fairly conservative settings that are safe for
syncing but that might not be the best performing one. Once you got
everything set up and running, you might want to look into speeding up
your synchronization. Here are a couple of hints and tips on how to
achieve this.
1) Use maxconnections > 1. By default we only use one connection to an
IMAP server. Using 2 or even 3 speeds things up considerably in most
cases. This setting goes into the [Repository XXX] section.
2) Use folderfilters. The quickest sync is a sync that can ignore some
folders. I sort my inbox into monthly folders, and ignore every
folder that is more than 2-3 months old, this lets me only inspect a
fraction of my Mails on every sync. If you haven't done this yet, do
it :). See the folderfilter section the example offlineimap.conf.
3) The default status cache is a plain text file that will write out
the complete file for each single new message (or even changed flag)
to a temporary file. If you have plenty of files in a folder, this
is a few hundred kilo to megabytes for each mail and is bound to
make things slower. I recommend to use the sqlite backend for
that. See the status_backend = sqlite setting in the example
offlineimap.conf. You will need to have python-sqlite installed in
order to use this. This will save you plenty of disk activity. Do
note that the sqlite backend is still considered experimental as it
has only been included recently (although a loss of your status
cache should not be a tragedy as that file can be rebuild
automatically)
4) Use quick sync. A regular sync will request all flags and all UIDs
of all mails in each folder which takes quite some time. A 'quick'
sync only compares the number of messages in a folder on the IMAP
side (it will detect flag changes on the Maildir side of things
though). A quick sync on my smallish account will take 7 seconds
rather than 40 seconds. Eg, I run a cron script that does a regular
sync once a day, and does quick syncs (-q) only synchronizing the
"-f INBOX" in between.
5) Turn off fsync. In the [general] section you can set fsync to True
or False. If you want to play 110% safe and wait for all operations
to hit the disk before continueing, you can set this to True. If you
set it to False, you lose some of that safety, trading it for speed.
Security and SSL
================
Some words on OfflineImap and its use of SSL/TLS. By default, we will
connect using any method that openssl supports, that is SSLv2, SSLv3, or
TLSv1. Do note that SSLv2 is notoriously insecure and deprecated.
Unfortunately, python2 does not offer easy ways to disable SSLv2. It is
recommended you test your setup and make sure that the mail server does
not use an SSLv2 connection. Use e.g. "openssl s_client -host
mail.server -port 443" to find out the connection that is used by
default.
Certificate checking
--------------------
Unfortunately, by default we will not verify the certificate of an IMAP
TLS/SSL server we connect to, so connecting by SSL is no guarantee
against man-in-the-middle attacks. While verifying a server certificate
fingerprint is being planned, it is not implemented yet. There is
currently only one safe way to ensure that you connect to the correct
server in an encrypted manner: You can specify a 'sslcacertfile' setting
in your repository section of offlineimap.conf pointing to a file that
contains (among others) a CA Certificate in PEM format which validating
your server certificate. In this case, we will check that: 1) The server
SSL certificate is validated by the CA Certificate 2) The server host
name matches the SSL certificate 3) The server certificate is not past
its expiration date. The FAQ contains an entry on how to create your own
certificate and CA certificate.
StartTLS
--------
If you have not configured your account to connect via SSL anyway,
OfflineImap will still attempt to set up an SSL connection via the
STARTTLS function, in case the imap server supports it. Do note, that
there is no certificate or fingerprint checking involved at all, when
using STARTTLS (the underlying imaplib library does not support this
yet). This means that you will be protected against passively listening
eavesdroppers and they will not be able to see your password or email
contents. However, this will not protect you from active attacks, such
as Man-In-The-Middle attacks which cause you to connect to the wrong
server and pretend to be your mail server. DO NOT RELY ON STARTTLS AS A
SAFE CONNECTION GUARANTEEING THE AUTHENTICITY OF YOUR IMAP SERVER!
UNIX Signals
============
OfflineImap listens to the unix signals SIGUSR1 and SIGUSR2. OfflineImap listens to the unix signals SIGUSR1 and SIGUSR2.
@ -310,7 +401,7 @@ KNOWN BUGS
storing messages. Such files can be written to windows partitions. But storing messages. Such files can be written to windows partitions. But
you will probably loose compatibility with other programs trying to you will probably loose compatibility with other programs trying to
read the same Maildir. read the same Maildir.
- Exclamation mark was choosed because of the note in - Exclamation mark was chosen because of the note in
http://docs.python.org/library/mailbox.html http://docs.python.org/library/mailbox.html
- If you have some messages already stored without this option, you will - If you have some messages already stored without this option, you will
have to re-sync them again have to re-sync them again
@ -322,92 +413,146 @@ KNOWN BUGS
- not available anymore since cygwin 1.7 - not available anymore since cygwin 1.7
Synchronization Performance PITFALLS & ISSUES
=========================== =================
By default, we use fairly conservative settings that are good for Sharing a maildir with multiple IMAP servers
syncing but that might not be the best performing one. Once you got --------------------------------------------
everything set up and running, you might want to look into speeding up
your synchronization. Here are a couple of hints and tips on how to
achieve this.
1) Use maxconnections > 1. By default we only use one connection to an Generally a word of caution mixing IMAP repositories on the same
IMAP server. Using 2 or even 3 speeds things up considerably in most Maildir root. You have to be careful that you *never* use the same
cases. This setting goes into the [Repository XXX] section. maildir folder for 2 IMAP servers. In the best case, the folder MD5
will be different, and you will get a loop where it will upload your
mails to both servers in turn (infinitely!) as it thinks you have
placed new mails in the local Maildir. In the worst case, the MD5 is
the same (likely) and mail UIDs overlap (likely too!) and it will fail to
sync some mails as it thinks they are already existent.
2) Use folderfilters. The quickest sync is a sync that can ignore some I would create a new local Maildir Repository for the Personal Gmail and
folders. I sort my inbox into monthly folders, and ignore every use a different root to be on the safe side here. You could e.g. use
folder that is more than 2-3 months old, this lets me only inspect a `~/mail/Pro` as Maildir root for the ProGmail and
fraction of my Mails on every sync. If you haven't done this yet, do `~/mail/Personal` as root for the personal one.
it :). See the folderfilter section the example offlineimap.conf.
3) The default status cache is a plain text file that will write out If you then point your local mutt, or whatever MUA you use to `~/mail/`
the complete file for each single new message (or even changed flag) as root, it should still recognize all folders. (see the 2 IMAP setup
to a temporary file. If you have plenty of files in a folder, this in the `Use Cases`_ section.
is a few hundred kilo to megabytes for each mail and is bound to
make things slower. I recommend to use the sqlite backend for
that. See the status_backend = sqlite setting in the example
offlineimap.conf. You will need to have python-sqlite installed in
order to use this. This will save you plenty of disk activity. Do
note that the sqlite backend is still considered experimental as it
has only been included recently (although a loss of your status
cache should not be a tragedy as that file can be rebuild
automatically)
4) Use quick sync. A regular sync will request all flags and all UIDs USE CASES
of all mails in each folder which takes quite some time. A 'quick' =========
sync only compares the number of messages in a folder on the IMAP
side (it will detect flag changes on the Maildir side of things
though). A quick sync on my smallish account will take 7 seconds
rather than 40 seconds. Eg, I run a cron script that does a regular
sync once a day, and does quick syncs inbetween.
5) Turn off fsync. In the [general] section you can set fsync to True Sync from GMail to another IMAP server
or False. If you want to play 110% safe and wait for all operations --------------------------------------
to hit the disk before continueing, you can set this to True. If you
set it to False, you lose some of that safety trading it for speed.
Security and SSL This is an example of a setup where "TheOtherImap" requires all folders to be under INBOX::
================
Some words on OfflineImap and its use of SSL/TLS. By default, we will [Repository Gmailserver-foo]
connect using any method that openssl supports, that is SSLv2, SSLv3, or #This is the remote repository
TLSv1. Do note that SSLv2 is notoriously insecure and deprecated. type = Gmail
Unfortunately, python2 does not offer easy ways to disable SSLv2. It is remotepass = XXX
recommended you test your setup and make sure that the mail server does remoteuser = XXX
not use an SSLv2 connection. Use e.g. "openssl s_client -host # The below will put all GMAIL folders as sub-folders of the 'local' INBOX,
mail.server -port 443" to find out the connection that is used by # assuming that your path separator on 'local' is a dot.
default. nametrans = lambda x: 'INBOX.' + x
Certificate checking [Repository TheOtherImap]
^^^^^^^^^^^^^^^^^^^^ #This is the 'local' repository
type = IMAP
remotehost = XXX
remotepass = XXX
remoteuser = XXX
#Do not use nametrans here.
Unfortunately, by default we will not verify the certificate of an IMAP Selecting only a few folders to sync
TLS/SSL server we connect to, so connecting by SSL is no guarantee ------------------------------------
against man-in-the-middle attacks. While verifying a server certificate Add this to the remote gmail repository section to only sync mails which are in a certain folder::
fingerprint is being planned, it is not implemented yet. There is
currently only one safe way to ensure that you connect to the correct
server in an encrypted manner: You can specify a 'sslcacertfile' setting
in your repository section of offlineimap.conf pointing to a file that
contains (among others) a CA Certificate in PEM format which validating
your server certificate. In this case, we will check that: 1) The server
SSL certificate is validated by the CA Certificate 2) The server host
name matches the SSL certificate 3) The server certificate is not past
its expiration date. The FAQ contains an entry on how to create your own
certificate and CA certificate.
StartTLS folderfilter = lambda folder: folder.startswith('MyLabel')
^^^^^^^^
If you have not configured your account to connect via SSL anyway, To only get the All Mail folder from a Gmail account, you would e.g. do::
OfflineImap will still attempt to set up an SSL connection via the
STARTTLS function, in case the imap server supports it. Do note, that folderfilter = lambda folder: folder.startswith('[Gmail]/All Mail')
there is no certificate or fingerprint checking involved at all, when
using STARTTLS (the underlying imaplib library does not support this
yet). This means that you will be protected against passively listening Another nametrans transpose example
eavesdroppers and they will not be able to see your password or email -----------------------------------
contents. However, this will not protect you from active attacks, such
as Man-In-The-Middle attacks which cause you to connect to the wrong Put everything in a GMX. subfolder except for the boxes INBOX, Draft, and Sent which should keep the same name::
server and pretend to be your mail server. DO NOT RELY ON STARTTLS AS A
SAFE CONNECTION GUARANTEEING THE AUTHENTICITY OF YOUR IMAP SERVER! nametrans: lambda folder: folder if folder in ['INBOX', 'Drafts', 'Sent'] \
======= else re.sub(r'^', r'GMX.', folder)
2 IMAP using name translations
------------------------------
Synchronizing 2 IMAP accounts to local Maildirs that are "next to each other", so that mutt can work on both. Full email setup described by Thomas Kahle at `http://dev.gentoo.org/~tomka/mail.html`_
offlineimap.conf::
[general]
accounts = acc1, acc2
maxsyncaccounts = 2
ui = ttyui
pythonfile=~/bin/offlineimap-helpers.py
socktimeout = 90
[Account acc1]
localrepository = acc1local
remoterepository = acc1remote
autorefresh = 2
[Account acc2]
localrepository = acc2local
remoterepository = acc2remote
autorefresh = 4
[Repository acc1local]
type = Maildir
localfolders = ~/Mail/acc1
[Repository acc2local]
type = Maildir
localfolders = ~/Mail/acc2
[Repository acc1remote]
type = IMAP
remotehost = imap.acc1.com
remoteusereval = get_username("imap.acc1.net")
remotepasseval = get_password("imap.acc1.net")
nametrans = oimaptransfolder_acc1
ssl = yes
maxconnections = 2
# Folders to get:
folderfilter = lambda foldername: foldername in [
'INBOX', 'Drafts', 'Sent', 'archiv']
[Repository acc2remote]
type = IMAP
remotehost = imap.acc2.net
remoteusereval = get_username("imap.acc2.net")
remotepasseval = get_password("imap.acc2.net")
nametrans = oimaptransfolder_acc2
ssl = yes
maxconnections = 2
One of the coolest things about offlineimap is that you can inject arbitrary python code. The file specified with::
pythonfile=~/bin/offlineimap-helpers.py
contains python functions that I used for two purposes: Fetching passwords from the gnome-keyring and translating folder names on the server to local foldernames. The python file should contain all the functions that are called here. get_username and get_password are part of the interaction with gnome-keyring and not printed here. Find them in the example file that is in the tarball or here. The folderfilter is a lambda term that, well, filters which folders to get. `oimaptransfolder_acc2` translates remote folders into local folders with a very simple logic. The `INBOX` folder will simply have the same name as the account while any other folder will have the account name and a dot as a prefix. offlineimap handles the renaming correctly in both directions::
import re
def oimaptransfolder_acc1(foldername):
if(foldername == "INBOX"):
retval = "acc1"
else:
retval = "acc1." + foldername
retval = re.sub("/", ".", retval)
return retval
def oimaptransfolder_acc2(foldername):
if(foldername == "INBOX"):
retval = "acc2"
else:
retval = "acc2." + foldername
retval = re.sub("/", ".", retval)
return retval

View File

@ -5,9 +5,9 @@
Welcome to :mod:`offlineimaps`'s documentation Welcome to :mod:`offlineimaps`'s documentation
============================================== ==============================================
The :mod:`offlineimap` module provides the user interface for synchronization between IMAP servers and MailDirs or between IMAP servers. The homepage containing the source code repository can be found at the `offlineimap homepage <http://offlineimap.org>`_. The :mod:`offlineimap` module provides the user interface for synchronization between IMAP servers and MailDirs or between IMAP servers. The homepage containing the source code repository can be found at the `offlineimap homepage <http://offlineimap.org>`_. The following provides the developer documentation for those who are interested in modifying the source code or otherwise peek into the OfflineImap internals. End users might want to check the MANUAL, our INSTALLation instructions, and the FAQ.
Within :mod:`offlineimap`, the classes :class:`OfflineImap` provides the high-level functionality. The rest of the classes should usually not needed to be touched by the user. A folder is represented by a :class:`offlineimap.folder.Base.BaseFolder` or any derivative :mod:`offlineimap.folder`. Within :mod:`offlineimap`, the classes :class:`OfflineImap` provides the high-level functionality. The rest of the classes should usually not needed to be touched by the user. Email repositories are represented by a :class:`offlineimap.repository.Base.BaseRepository` or derivatives (see :mod:`offlineimap.repository` for details). A folder within a repository is represented by a :class:`offlineimap.folder.Base.BaseFolder` or any derivative from :mod:`offlineimap.folder`.
.. moduleauthor:: John Goerzen, and many others. See AUTHORS and the git history for a full list. .. moduleauthor:: John Goerzen, and many others. See AUTHORS and the git history for a full list.
@ -15,7 +15,7 @@ Within :mod:`offlineimap`, the classes :class:`OfflineImap` provides the high-le
This page contains the main API overview of OfflineImap |release|. This page contains the main API overview of OfflineImap |release|.
Notmuch can be imported as:: OfflineImap can be imported as::
from offlineimap import OfflineImap from offlineimap import OfflineImap
@ -58,11 +58,12 @@ An :class:`accounts.Account` connects two email repositories that are to be sync
Contains the current :mod:`offlineimap.ui`, and can be used for logging etc. Contains the current :mod:`offlineimap.ui`, and can be used for logging etc.
:exc:`OfflineImapError` -- A Notmuch execution error
:exc:`OfflineImapException` -- A Notmuch execution error
-------------------------------------------------------- --------------------------------------------------------
.. autoexception:: offlineimap.OfflineImapException .. autoexception:: offlineimap.error.OfflineImapError
:members: :members:
This execption inherits directly from :exc:`Exception` and is raised on errors during the offlineimap execution. This execption inherits directly from :exc:`Exception` and is raised
on errors during the offlineimap execution. It has an attribute
`severity` that denotes the severity level of the error.

View File

@ -25,8 +25,8 @@ on your configuration. So when you want to instanciate a new
:mod:`offlineimap.repository` -- Basic representation of a mail repository :mod:`offlineimap.repository.Base.BaseRepository` -- Representation of a mail repository
-------------------------------------------------------------------------- ------------------------------------------------------------------------------------------
.. autoclass:: offlineimap.repository.Base.BaseRepository .. autoclass:: offlineimap.repository.Base.BaseRepository
:members: :members:
:inherited-members: :inherited-members:

View File

@ -1,4 +1,4 @@
:mod:`offlineimap.ui` -- A pluggable logging system :mod:`offlineimap.ui` -- A flexible logging system
-------------------------------------------------------- --------------------------------------------------------
.. currentmodule:: offlineimap.ui .. currentmodule:: offlineimap.ui

View File

@ -15,8 +15,26 @@
# along with this program; if not, write to the Free Software # along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
# This file documents all possible options and can be quite scary.
# Looking for a quick start? Take a look at offlineimap.conf.minimal. # Looking for a quick start? Take a look at offlineimap.conf.minimal.
# Settings support interpolation. This means values can contain python
# format strings which refer to other values in the same section, or
# values in a special DEFAULT section. This allows you for example to
# use common settings for multiple accounts:
#
# [Repository Gmail1]
# trashfolder: %(gmailtrashfolder)s
#
# [Repository Gmail2]
# trashfolder: %(gmailtrashfolder)s
#
# [DEFAULT]
# gmailtrashfolder = [Google Mail]/Papierkorb
#
# would set the trashfolder setting for your German gmail accounts.
################################################## ##################################################
# General definitions # General definitions
@ -141,10 +159,6 @@ footer = "\n"
# Note that this filter can be used only to further restrict mbnames # Note that this filter can be used only to further restrict mbnames
# to a subset of folders that pass the account's folderfilter. # to a subset of folders that pass the account's folderfilter.
[ui.Curses.Blinkenlights]
# Character used to indicate thread status.
statuschar = .
################################################## ##################################################
# Accounts # Accounts
@ -205,8 +219,7 @@ remoterepository = RemoteExample
# state in plain text files. On Repositories with large numbers of # state in plain text files. On Repositories with large numbers of
# mails, the performance might not be optimal, as we write out the # mails, the performance might not be optimal, as we write out the
# complete file for each change. Another new backend 'sqlite' is # complete file for each change. Another new backend 'sqlite' is
# available which stores the status in sqlite databases. BE AWARE THIS # available which stores the status in sqlite databases.
# IS EXPERIMENTAL STUFF.
# #
# If you switch the backend, you may want to delete the old cache # If you switch the backend, you may want to delete the old cache
# directory in ~/.offlineimap/Account-<account>/LocalStatus manually # directory in ~/.offlineimap/Account-<account>/LocalStatus manually
@ -314,6 +327,16 @@ ssl = yes
# The certificate should be in PEM format. # The certificate should be in PEM format.
# sslcacertfile = /path/to/cacertfile.crt # sslcacertfile = /path/to/cacertfile.crt
# If you connect via SSL/TLS (ssl=true) and you have no CA certificate
# specified, offlineimap will refuse to sync as it connects to a server
# with an unknown "fingerprint". If you are sure you connect to the
# correct server, you can then configure the presented server
# fingerprint here. OfflineImap will verify that the server fingerprint
# has not changed on each connect and refuse to connect otherwise.
# You can also configure this in addition to CA certificate validation
# above and it will check both ways. cert_fingerprint =
# <SHA1_of_server_certificate_here>
# Specify the port. If not specified, use a default port. # Specify the port. If not specified, use a default port.
# remoteport = 993 # remoteport = 993
@ -383,8 +406,8 @@ remoteuser = username
# holdconnectionopen - to be true # holdconnectionopen - to be true
# keepalive - to be 29 minutes unless you specify otherwise # keepalive - to be 29 minutes unless you specify otherwise
# #
# This feature isn't complete and may well have problems. BE AWARE THIS # This feature isn't complete and may well have problems. See the manual
# IS EXPERIMENTAL STUFF. See the manual for more details. # for more details.
# #
# This option should return a Python list. For example # This option should return a Python list. For example
# #
@ -447,10 +470,12 @@ subscribedonly = no
# #
# nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername) # nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)
# You can specify which folders to sync. You can do it several ways. # You can specify which folders to sync using the folderfilter
# I'll provide some examples. The folderfilter operates on the # setting. You can provide any python function (e.g. a lambda function)
# *UNTRANSLATED* name, if you specify nametrans. It should return # which will be invoked for each foldername. If the filter function
# true if the folder is to be included; false otherwise. # returns True, the folder will be synced, if it returns False, it. The
# folderfilter operates on the *UNTRANSLATED* name (before any nametrans
# translation takes place).
# #
# Example 1: synchronizing only INBOX and Sent. # Example 1: synchronizing only INBOX and Sent.
# #
@ -474,34 +499,17 @@ subscribedonly = no
# folderfilter = lambda foldername: foldername in # folderfilter = lambda foldername: foldername in
# ['INBOX', 'Sent Mail', 'Deleted Items', # ['INBOX', 'Sent Mail', 'Deleted Items',
# 'Received'] # 'Received']
#
# FYI, you could also include every folder with:
#
# folderfilter = lambda foldername: 1
#
# And exclude every folder with:
#
# folderfilter = lambda foldername: 0
# You can specify folderincludes to include additional folders.
# It should return a Python list. This might be used to include a
# folder that was excluded by your folderfilter rule, to include a # You can specify folderincludes to include additional folders. It
# folder that your server does not specify with its LIST option, or # should return a Python list. This might be used to include a folder
# to include a folder that is outside your basic reference. Some examples: # that was excluded by your folderfilter rule, to include a folder that
# # your server does not specify with its LIST option, or to include a
# To include debian.user and debian.personal: # folder that is outside your basic reference. The 'reference' value
# # will not be prefixed to this folder name, even if you have specified
# one. For example:
# folderincludes = ['debian.user', 'debian.personal'] # folderincludes = ['debian.user', 'debian.personal']
#
# To include your INBOX (UW IMAPd users will find this useful if they
# specify a reference):
#
# folderincludes = ['INBOX']
#
# To specify a long list:
#
# folderincludes = ['box1', 'box2', 'box3', 'box4',
# 'box5', 'box6']
# You can specify foldersort to determine how folders are sorted. # You can specify foldersort to determine how folders are sorted.
# This affects order of synchronization and mbnames. The expression # This affects order of synchronization and mbnames. The expression

View File

@ -15,11 +15,11 @@
# along with this program; if not, write to the Free Software # along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
from ConfigParser import ConfigParser from ConfigParser import SafeConfigParser
from offlineimap.localeval import LocalEval from offlineimap.localeval import LocalEval
import os import os
class CustomConfigParser(ConfigParser): class CustomConfigParser(SafeConfigParser):
def getdefault(self, section, option, default, *args, **kwargs): def getdefault(self, section, option, default, *args, **kwargs):
"""Same as config.get, but returns the "default" option if there """Same as config.get, but returns the "default" option if there
is no such option specified.""" is no such option specified."""

View File

@ -1,7 +1,7 @@
__all__ = ['OfflineImap'] __all__ = ['OfflineImap']
__productname__ = 'OfflineIMAP' __productname__ = 'OfflineIMAP'
__version__ = "6.3.4" __version__ = "6.3.5-rc1"
__copyright__ = "Copyright 2002-2011 John Goerzen & contributors" __copyright__ = "Copyright 2002-2011 John Goerzen & contributors"
__author__ = "John Goerzen" __author__ = "John Goerzen"
__author_email__= "john@complete.org" __author_email__= "john@complete.org"

View File

@ -22,6 +22,7 @@ from offlineimap.threadutil import InstanceLimitedThread
from subprocess import Popen, PIPE from subprocess import Popen, PIPE
from threading import Event from threading import Event
import os import os
from sys import exc_info
import traceback import traceback
def getaccountlist(customconfig): def getaccountlist(customconfig):
@ -178,16 +179,16 @@ class SyncableAccount(Account):
except (KeyboardInterrupt, SystemExit): except (KeyboardInterrupt, SystemExit):
raise raise
except OfflineImapError, e: except OfflineImapError, e:
self.ui.warn(e.reason)
# Stop looping and bubble up Exception if needed. # Stop looping and bubble up Exception if needed.
if e.severity >= OfflineImapError.ERROR.REPO: if e.severity >= OfflineImapError.ERROR.REPO:
if looping: if looping:
looping -= 1 looping -= 1
if e.severity >= OfflineImapError.ERROR.CRITICAL: if e.severity >= OfflineImapError.ERROR.CRITICAL:
raise raise
except: self.ui.error(e, exc_info()[2])
self.ui.warn("Error occured attempting to sync account "\ except Exception, e:
"'%s':\n%s"% (self, traceback.format_exc())) self.ui.error(e, msg = "While attempting to sync "
"account %s:\n %s"% (self, traceback.format_exc()))
else: else:
# after success sync, reset the looping counter to 3 # after success sync, reset the looping counter to 3
if self.refreshperiod: if self.refreshperiod:
@ -232,7 +233,7 @@ class SyncableAccount(Account):
# replicate the folderstructure from REMOTE to LOCAL # replicate the folderstructure from REMOTE to LOCAL
if not localrepos.getconf('readonly', False): if not localrepos.getconf('readonly', False):
self.ui.syncfolders(remoterepos, localrepos) self.ui.syncfolders(remoterepos, localrepos)
remoterepos.syncfoldersto(localrepos, [statusrepos]) remoterepos.syncfoldersto(localrepos, statusrepos)
# iterate through all folders on the remote repo and sync # iterate through all folders on the remote repo and sync
for remotefolder in remoterepos.getfolders(): for remotefolder in remoterepos.getfolders():
@ -276,8 +277,10 @@ class SyncableAccount(Account):
r = p.communicate() r = p.communicate()
self.ui.callhook("Hook stdout: %s\nHook stderr:%s\n" % r) self.ui.callhook("Hook stdout: %s\nHook stderr:%s\n" % r)
self.ui.callhook("Hook return code: %d" % p.returncode) self.ui.callhook("Hook return code: %d" % p.returncode)
except: except (KeyboardInterrupt, SystemExit):
self.ui.warn("Exception occured while calling hook") raise
except Exception, e:
self.ui.error(e, exc_info()[2], msg = "Calling hook")
def syncfolder(accountname, remoterepos, remotefolder, localrepos, def syncfolder(accountname, remoterepos, remotefolder, localrepos,
@ -366,9 +369,9 @@ def syncfolder(accountname, remoterepos, remotefolder, localrepos,
if e.severity > OfflineImapError.ERROR.FOLDER: if e.severity > OfflineImapError.ERROR.FOLDER:
raise raise
else: else:
ui.warn("Aborting folder sync '%s' [acc: '%s']\nReason was: %s" %\ ui.error(e, exc_info()[2], msg = "Aborting folder sync '%s' "
(localfolder.name, accountname, e.reason)) "[acc: '%s']" % (localfolder, accountname))
except: except Exception, e:
ui.warn("ERROR in syncfolder for %s folder %s: %s" % \ ui.error(e, msg = "ERROR in syncfolder for %s folder %s: %s" % \
(accountname,remotefolder.getvisiblename(), (accountname,remotefolder.getvisiblename(),
traceback.format_exc())) traceback.format_exc()))

View File

@ -2,11 +2,15 @@ class OfflineImapError(Exception):
"""An Error during offlineimap synchronization""" """An Error during offlineimap synchronization"""
class ERROR: class ERROR:
"""Severity levels""" """Severity level of an Exception
MESSAGE = 0
FOLDER = 10 * **MESSAGE**: Abort the current message, but continue with folder
REPO = 20 * **FOLDER_RETRY**: Error syncing folder, but do retry
CRITICAL = 30 * **FOLDER**: Abort folder sync, but continue with next folder
* **REPO**: Abort repository sync, continue with next account
* **CRITICAL**: Immediately exit offlineimap
"""
MESSAGE, FOLDER_RETRY, FOLDER, REPO, CRITICAL = 0, 10, 15, 20, 30
def __init__(self, reason, severity, errcode=None): def __init__(self, reason, severity, errcode=None):
""" """

View File

@ -17,9 +17,15 @@
from offlineimap import threadutil from offlineimap import threadutil
from offlineimap.ui import getglobalui from offlineimap.ui import getglobalui
from offlineimap.error import OfflineImapError
import os.path import os.path
import re import re
from sys import exc_info
import traceback import traceback
try: # python 2.6 has set() built in
set
except NameError:
from sets import Set as set
class BaseFolder(object): class BaseFolder(object):
def __init__(self): def __init__(self):
@ -70,11 +76,15 @@ class BaseFolder(object):
return self.getname() return self.getname()
def getfolderbasename(self): def getfolderbasename(self):
foldername = self.getname() """Return base file name of file to store Status/UID info in"""
foldername = foldername.replace(self.repository.getsep(), '.') if not self.name:
foldername = re.sub('/\.$', '/dot', foldername) basename = '.'
foldername = re.sub('^\.$', 'dot', foldername) else: #avoid directory hierarchies and file names such as '/'
return foldername basename = self.name.replace('/', '.')
# replace with literal 'dot' if final path name is '.' as '.' is
# an invalid file name.
basename = re.sub('(^|\/)\.$','\\1dot', basename)
return basename
def isuidvalidityok(self): def isuidvalidityok(self):
"""Does the cached UID match the real UID """Does the cached UID match the real UID
@ -183,12 +193,9 @@ class BaseFolder(object):
def addmessageflags(self, uid, flags): def addmessageflags(self, uid, flags):
"""Adds the specified flags to the message's flag set. If a given """Adds the specified flags to the message's flag set. If a given
flag is already present, it will not be duplicated.""" flag is already present, it will not be duplicated.
newflags = self.getmessageflags(uid) :param flags: A set() of flags"""
for flag in flags: newflags = self.getmessageflags(uid) | flags
if not flag in newflags:
newflags.append(flag)
newflags.sort()
self.savemessageflags(uid, newflags) self.savemessageflags(uid, newflags)
def addmessagesflags(self, uidlist, flags): def addmessagesflags(self, uidlist, flags):
@ -198,11 +205,7 @@ class BaseFolder(object):
def deletemessageflags(self, uid, flags): def deletemessageflags(self, uid, flags):
"""Removes each flag given from the message's flag set. If a given """Removes each flag given from the message's flag set. If a given
flag is already removed, no action will be taken for that flag.""" flag is already removed, no action will be taken for that flag."""
newflags = self.getmessageflags(uid) newflags = self.getmessageflags(uid) - flags
for flag in flags:
if flag in newflags:
newflags.remove(flag)
newflags.sort()
self.savemessageflags(uid, newflags) self.savemessageflags(uid, newflags)
def deletemessagesflags(self, uidlist, flags): def deletemessagesflags(self, uidlist, flags):
@ -229,10 +232,10 @@ class BaseFolder(object):
# synced to the status cache. This is only a problem with # synced to the status cache. This is only a problem with
# self.getmessage(). So, don't call self.getmessage unless # self.getmessage(). So, don't call self.getmessage unless
# really needed. # really needed.
try:
if register: # output that we start a new thread if register: # output that we start a new thread
self.ui.registerthread(self.getaccountname()) self.ui.registerthread(self.getaccountname())
try:
message = None message = None
flags = self.getmessageflags(uid) flags = self.getmessageflags(uid)
rtime = self.getmessagetime(uid) rtime = self.getmessagetime(uid)
@ -242,17 +245,17 @@ class BaseFolder(object):
statusfolder.savemessage(uid, None, flags, rtime) statusfolder.savemessage(uid, None, flags, rtime)
return return
self.ui.copyingmessage(uid, self, [dstfolder]) self.ui.copyingmessage(uid, self, dstfolder)
# If any of the destinations actually stores the message body, # If any of the destinations actually stores the message body,
# load it up. # load it up.
if dstfolder.storesmessages(): if dstfolder.storesmessages():
message = self.getmessage(uid) message = self.getmessage(uid)
#Succeeded? -> IMAP actually assigned a UID. If newid #Succeeded? -> IMAP actually assigned a UID. If newid
#remained negative, no server was willing to assign us an #remained negative, no server was willing to assign us an
#UID. If newid is 0, saving succeeded, but we could not #UID. If newid is 0, saving succeeded, but we could not
#retrieve the new UID. Ignore message in this case. #retrieve the new UID. Ignore message in this case.
newuid = dstfolder.savemessage(uid, message, flags, rtime) newuid = dstfolder.savemessage(uid, message, flags, rtime)
if newuid > 0: if newuid > 0:
if newuid != uid: if newuid != uid:
# Got new UID, change the local uid. # Got new UID, change the local uid.
@ -264,6 +267,7 @@ class BaseFolder(object):
uid = newuid uid = newuid
# Save uploaded status in the statusfolder # Save uploaded status in the statusfolder
statusfolder.savemessage(uid, message, flags, rtime) statusfolder.savemessage(uid, message, flags, rtime)
elif newuid == 0: elif newuid == 0:
# Message was stored to dstfolder, but we can't find it's UID # Message was stored to dstfolder, but we can't find it's UID
# This means we can't link current message to the one created # This means we can't link current message to the one created
@ -273,18 +277,22 @@ class BaseFolder(object):
# IMAP servers ... # IMAP servers ...
self.deletemessage(uid) self.deletemessage(uid)
else: else:
raise UserWarning("Trying to save msg (uid %d) on folder " raise OfflineImapError("Trying to save msg (uid %d) on folder "
"%s returned invalid uid %d" % \ "%s returned invalid uid %d" % \
(uid, (uid,
dstfolder.getvisiblename(), dstfolder.getvisiblename(),
newuid)) newuid),
except (KeyboardInterrupt): OfflineImapError.ERROR.MESSAGE)
raise except OfflineImapError, e:
except: if e.severity > OfflineImapError.ERROR.MESSAGE:
self.ui.warn("ERROR attempting to copy message " + str(uid) \ raise # buble severe errors up
+ " for account " + self.getaccountname() + ":" \ self.ui.error(e, exc_info()[2])
+ traceback.format_exc()) except Exception, e:
raise self.ui.error(e, "Copying message %s [acc: %s]:\n %s" %\
(uid, self.getaccountname(),
traceback.format_exc()))
raise #raise on unknown errors, so we can fix those
def syncmessagesto_copy(self, dstfolder, statusfolder): def syncmessagesto_copy(self, dstfolder, statusfolder):
"""Pass1: Copy locally existing messages not on the other side """Pass1: Copy locally existing messages not on the other side
@ -303,6 +311,7 @@ class BaseFolder(object):
statusfolder.uidexists(uid), statusfolder.uidexists(uid),
self.getmessageuidlist()) self.getmessageuidlist())
for uid in copylist: for uid in copylist:
# exceptions are caught in copymessageto()
if self.suggeststhreads(): if self.suggeststhreads():
self.waitforthread() self.waitforthread()
thread = threadutil.InstanceLimitedThread(\ thread = threadutil.InstanceLimitedThread(\
@ -315,8 +324,8 @@ class BaseFolder(object):
thread.start() thread.start()
threads.append(thread) threads.append(thread)
else: else:
self.copymessageto(uid, dstfolder, statusfolder, register = 0) self.copymessageto(uid, dstfolder, statusfolder,
register = 0)
for thread in threads: for thread in threads:
thread.join() thread.join()
@ -350,8 +359,8 @@ class BaseFolder(object):
addflaglist = {} addflaglist = {}
delflaglist = {} delflaglist = {}
for uid in self.getmessageuidlist(): for uid in self.getmessageuidlist():
# Ignore messages with negative UIDs missed by pass 1 # Ignore messages with negative UIDs missed by pass 1 and
# also don't do anything if the message has been deleted remotely # don't do anything if the message has been deleted remotely
if uid < 0 or not dstfolder.uidexists(uid): if uid < 0 or not dstfolder.uidexists(uid):
continue continue
@ -359,29 +368,30 @@ class BaseFolder(object):
statusflags = statusfolder.getmessageflags(uid) statusflags = statusfolder.getmessageflags(uid)
#if we could not get message flags from LocalStatus, assume empty. #if we could not get message flags from LocalStatus, assume empty.
if statusflags is None: if statusflags is None:
statusflags = [] statusflags = set()
addflags = [x for x in selfflags if x not in statusflags]
addflags = selfflags - statusflags
delflags = statusflags - selfflags
for flag in addflags: for flag in addflags:
if not flag in addflaglist: if not flag in addflaglist:
addflaglist[flag] = [] addflaglist[flag] = []
addflaglist[flag].append(uid) addflaglist[flag].append(uid)
delflags = [x for x in statusflags if x not in selfflags]
for flag in delflags: for flag in delflags:
if not flag in delflaglist: if not flag in delflaglist:
delflaglist[flag] = [] delflaglist[flag] = []
delflaglist[flag].append(uid) delflaglist[flag].append(uid)
for flag in addflaglist.keys(): for flag, uids in addflaglist.items():
self.ui.addingflags(addflaglist[flag], flag, dstfolder) self.ui.addingflags(uids, flag, dstfolder)
dstfolder.addmessagesflags(addflaglist[flag], [flag]) dstfolder.addmessagesflags(uids, set(flag))
statusfolder.addmessagesflags(addflaglist[flag], [flag]) statusfolder.addmessagesflags(uids, set(flag))
for flag in delflaglist.keys(): for flag,uids in delflaglist.items():
self.ui.deletingflags(delflaglist[flag], flag, dstfolder) self.ui.deletingflags(uids, flag, dstfolder)
dstfolder.deletemessagesflags(delflaglist[flag], [flag]) dstfolder.deletemessagesflags(uids, set(flag))
statusfolder.deletemessagesflags(delflaglist[flag], [flag]) statusfolder.deletemessagesflags(uids, set(flag))
def syncmessagesto(self, dstfolder, statusfolder): def syncmessagesto(self, dstfolder, statusfolder):
"""Syncs messages in this folder to the destination dstfolder. """Syncs messages in this folder to the destination dstfolder.
@ -421,8 +431,11 @@ class BaseFolder(object):
action(dstfolder, statusfolder) action(dstfolder, statusfolder)
except (KeyboardInterrupt): except (KeyboardInterrupt):
raise raise
except: except OfflineImapError, e:
self.ui.warn("ERROR attempting to sync flags " \ if e.severity > OfflineImapError.ERROR.FOLDER:
+ "for account " + self.getaccountname() \
+ ":" + traceback.format_exc())
raise raise
self.ui.error(e, exc_info()[2])
except Exception, e:
self.ui.error(e, exc_info()[2], "Syncing folder %s [acc: %s]" %\
(self, self.getaccountname()))
raise # raise unknown Exceptions so we can fix them

View File

@ -21,7 +21,6 @@
from IMAP import IMAPFolder from IMAP import IMAPFolder
from offlineimap import imaputil from offlineimap import imaputil
from copy import copy
class GmailFolder(IMAPFolder): class GmailFolder(IMAPFolder):
@ -55,7 +54,7 @@ class GmailFolder(IMAPFolder):
try: try:
imapobj.select(self.getfullname()) imapobj.select(self.getfullname())
result = imapobj.uid('copy', result = imapobj.uid('copy',
imaputil.listjoin(uidlist), imaputil.uid_sequence(uidlist),
self.trash_folder) self.trash_folder)
assert result[0] == 'OK', \ assert result[0] == 'OK', \
"Bad IMAPlib result: %s" % result[0] "Bad IMAPlib result: %s" % result[0]
@ -65,57 +64,3 @@ class GmailFolder(IMAPFolder):
del self.messagelist[uid] del self.messagelist[uid]
else: else:
IMAPFolder.deletemessages_noconvert(self, uidlist) IMAPFolder.deletemessages_noconvert(self, uidlist)
def processmessagesflags(self, operation, uidlist, flags):
# XXX: the imapobj.myrights(...) calls dies with an error
# report from Gmail server stating that IMAP command
# 'MYRIGHTS' is not implemented. So, this
# `processmessagesflags` is just a copy from `IMAPFolder`,
# with the references to `imapobj.myrights()` deleted This
# shouldn't hurt, however, Gmail users always have full
# control over all their mailboxes (apparently).
if len(uidlist) > 101:
# Hack for those IMAP ervers with a limited line length
self.processmessagesflags(operation, uidlist[:100], flags)
self.processmessagesflags(operation, uidlist[100:], flags)
return
imapobj = self.imapserver.acquireconnection()
try:
imapobj.select(self.getfullname())
r = imapobj.uid('store',
imaputil.listjoin(uidlist),
operation + 'FLAGS',
imaputil.flagsmaildir2imap(flags))
assert r[0] == 'OK', 'Error with store: ' + '. '.join(r[1])
r = r[1]
finally:
self.imapserver.releaseconnection(imapobj)
needupdate = copy(uidlist)
for result in r:
if result == None:
# Compensate for servers that don't return anything from
# STORE.
continue
attributehash = imaputil.flags2hash(imaputil.imapsplit(result)[1])
if not ('UID' in attributehash and 'FLAGS' in attributehash):
# Compensate for servers that don't return a UID attribute.
continue
flags = attributehash['FLAGS']
uid = long(attributehash['UID'])
self.messagelist[uid]['flags'] = imaputil.flagsimap2maildir(flags)
try:
needupdate.remove(uid)
except ValueError: # Let it slide if it's not in the list
pass
for uid in needupdate:
if operation == '+':
for flag in flags:
if not flag in self.messagelist[uid]['flags']:
self.messagelist[uid]['flags'].append(flag)
self.messagelist[uid]['flags'].sort()
elif operation == '-':
for flag in flags:
if flag in self.messagelist[uid]['flags']:
self.messagelist[uid]['flags'].remove(flag)

View File

@ -21,9 +21,15 @@ import random
import binascii import binascii
import re import re
import time import time
from copy import copy from sys import exc_info
from Base import BaseFolder from Base import BaseFolder
from offlineimap import imaputil, imaplibutil, OfflineImapError from offlineimap import imaputil, imaplibutil, OfflineImapError
from offlineimap.imaplib2 import MonthNames
try: # python 2.6 has set() built in
set
except NameError:
from sets import Set as set
class IMAPFolder(BaseFolder): class IMAPFolder(BaseFolder):
def __init__(self, imapserver, name, visiblename, accountname, repository): def __init__(self, imapserver, name, visiblename, accountname, repository):
@ -105,78 +111,75 @@ class IMAPFolder(BaseFolder):
self.imapserver.releaseconnection(imapobj) self.imapserver.releaseconnection(imapobj)
return False return False
# TODO: Make this so that it can define a date that would be the oldest messages etc.
def cachemessagelist(self): def cachemessagelist(self):
imapobj = self.imapserver.acquireconnection() maxage = self.config.getdefaultint("Account %s" % self.accountname,
"maxage", -1)
maxsize = self.config.getdefaultint("Account %s" % self.accountname,
"maxsize", -1)
self.messagelist = {} self.messagelist = {}
imapobj = self.imapserver.acquireconnection()
try: try:
# Primes untagged_responses res_type, imapdata = imapobj.select(self.getfullname(), True)
imaptype, imapdata = imapobj.select(self.getfullname(), readonly = 1, force = 1) if imapdata == [None] or imapdata[0] == '0':
# Empty folder, no need to populate message list
maxage = self.config.getdefaultint("Account " + self.accountname, "maxage", -1) return
maxsize = self.config.getdefaultint("Account " + self.accountname, "maxsize", -1) # By default examine all UIDs in this folder
msgsToFetch = '1:*'
if (maxage != -1) | (maxsize != -1): if (maxage != -1) | (maxsize != -1):
try: search_cond = "(";
search_condition = "(";
if(maxage != -1): if(maxage != -1):
#find out what the oldest message is that we should look at #find out what the oldest message is that we should look at
oldest_time_struct = time.gmtime(time.time() - (60*60*24*maxage)) oldest_struct = time.gmtime(time.time() - (60*60*24*maxage))
if oldest_struct[0] < 1900:
#format this manually - otherwise locales could cause problems raise OfflineImapError("maxage setting led to year %d. "
monthnames_standard = ["Jan", "Feb", "Mar", "Apr", "May", \ "Abort syncing." % oldest_struct[0],
"Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"] OfflineImapError.ERROR.REPO)
search_cond += "SINCE %02d-%s-%d" % (
our_monthname = monthnames_standard[oldest_time_struct[1]-1] oldest_struct[2],
daystr = "%(day)02d" % {'day' : oldest_time_struct[2]} MonthNames[oldest_struct[1]],
date_search_str = "SINCE " + daystr + "-" + our_monthname \ oldest_struct[0])
+ "-" + str(oldest_time_struct[0])
search_condition += date_search_str
if(maxsize != -1): if(maxsize != -1):
if(maxage != -1): #There are two conditions - add a space if(maxage != -1): # There are two conditions, add space
search_condition += " " search_cond += " "
search_cond += "SMALLER %d" % maxsize
search_condition += "SMALLER " + self.config.getdefault("Account " + self.accountname, "maxsize", -1) search_cond += ")"
search_condition += ")" res_type, res_data = imapobj.search(None, search_cond)
searchresult = imapobj.search(None, search_condition) if res_type != 'OK':
raise OfflineImapError("SEARCH in folder [%s]%s failed. "
"Search string was '%s'. Server responded '[%s] %s'" % (
self.getrepository(), self,
search_cond, res_type, res_data),
OfflineImapError.ERROR.FOLDER)
#result would come back seperated by space - to change into a fetch # Result UIDs are seperated by space, coalesce into ranges
#statement we need to change space to comma msgsToFetch = imaputil.uid_sequence(res_data[0].split())
messagesToFetch = searchresult[1][0].replace(" ", ",") if not msgsToFetch:
except KeyError: return # No messages to sync
return
if len(messagesToFetch) < 1:
# No messages; return
return
else:
# 1. Some mail servers do not return an EXISTS response
# if the folder is empty. 2. ZIMBRA servers can return
# multiple EXISTS replies in the form 500, 1000, 1500,
# 1623 so check for potentially multiple replies.
if imapdata == [None]:
return
maxmsgid = 0 # Get the flags and UIDs for these. single-quotes prevent
for msgid in imapdata: # imaplib2 from quoting the sequence.
maxmsgid = max(long(msgid), maxmsgid) res_type, response = imapobj.fetch("'%s'" % msgsToFetch,
if maxmsgid < 1: '(FLAGS UID)')
#no messages; return if res_type != 'OK':
return raise OfflineImapError("FETCHING UIDs in folder [%s]%s failed. "
messagesToFetch = '1:%d' % maxmsgid; "Server responded '[%s] %s'" % (
self.getrepository(), self,
# Now, get the flags and UIDs for these. res_type, response),
# We could conceivably get rid of maxmsgid and just say OfflineImapError.ERROR.FOLDER)
# '1:*' here.
response = imapobj.fetch(messagesToFetch, '(FLAGS UID)')[1]
finally: finally:
self.imapserver.releaseconnection(imapobj) self.imapserver.releaseconnection(imapobj)
for messagestr in response: for messagestr in response:
# Discard the message number. # looks like: '1 (FLAGS (\\Seen Old) UID 4807)' or None if no msg
# Discard initial message number.
if messagestr == None:
continue
messagestr = messagestr.split(' ', 1)[1] messagestr = messagestr.split(' ', 1)[1]
options = imaputil.flags2hash(messagestr) options = imaputil.flags2hash(messagestr)
if not options.has_key('UID'): if not options.has_key('UID'):
@ -200,14 +203,28 @@ class IMAPFolder(BaseFolder):
this UID could be found. this UID could be found.
""" """
imapobj = self.imapserver.acquireconnection() imapobj = self.imapserver.acquireconnection()
try:
fails_left = 2 # retry on dropped connection
while fails_left:
try: try:
imapobj.select(self.getfullname(), readonly = 1) imapobj.select(self.getfullname(), readonly = 1)
res_type, data = imapobj.uid('fetch', str(uid), '(BODY.PEEK[])') res_type, data = imapobj.uid('fetch', str(uid),
'(BODY.PEEK[])')
fails_left = 0
except imapobj.abort(), e:
# Release dropped connection, and get a new one
self.imapserver.releaseconnection(imapobj)
imapobj = self.imapserver.acquireconnection()
self.ui.error(e, exc_info()[2])
fails_left -= 1
if not fails_left:
raise e
if data == [None] or res_type != 'OK': if data == [None] or res_type != 'OK':
#IMAP server says bad request or UID does not exist #IMAP server says bad request or UID does not exist
severity = OfflineImapError.ERROR.MESSAGE severity = OfflineImapError.ERROR.MESSAGE
reason = "IMAP server '%s' responded with '%s' to fetching "\ reason = "IMAP server '%s' failed to fetch message UID '%d'."\
"message UID '%d'" % (self.getrepository(), res_type, uid) "Server responded: %s %s" % (self.getrepository(), uid,
res_type, data)
if data == [None]: if data == [None]:
#IMAP server did not find a message with this UID #IMAP server did not find a message with this UID
reason = "IMAP server '%s' does not have a message "\ reason = "IMAP server '%s' does not have a message "\
@ -216,16 +233,6 @@ class IMAPFolder(BaseFolder):
# data looks now e.g. [('320 (UID 17061 BODY[] # data looks now e.g. [('320 (UID 17061 BODY[]
# {2565}','msgbody....')] we only asked for one message, # {2565}','msgbody....')] we only asked for one message,
# and that msg is in data[0]. msbody is in [0][1] # and that msg is in data[0]. msbody is in [0][1]
#NB & TODO: When the message on the IMAP server has been
#deleted in the mean time, it will respond with an 'OK'
#res_type, but it will simply not send any data. This will
#lead to a crash in the below line. We need urgently to
#detect this, protect from this and need to think about what
#to return in this case. Probably returning `None` in this
#case would be good. But we need to make sure that all
#Backends behave the same, and that we actually check the
#return value and behave accordingly.
data = data[0][1].replace("\r\n", "\n") data = data[0][1].replace("\r\n", "\n")
if len(data)>200: if len(data)>200:
@ -317,6 +324,74 @@ class IMAPFolder(BaseFolder):
matchinguids.sort() matchinguids.sort()
return long(matchinguids[0]) return long(matchinguids[0])
def savemessage_fetchheaders(self, imapobj, headername, headervalue):
""" We fetch all new mail headers and search for the right
X-OfflineImap line by hand. The response from the server has form:
(
'OK',
[
(
'185 (RFC822.HEADER {1789}',
'... mail headers ...'
),
' UID 2444)',
(
'186 (RFC822.HEADER {1789}',
'... 2nd mail headers ...'
),
' UID 2445)'
]
)
We need to locate the UID just after mail headers containing our
X-OfflineIMAP line.
Returns UID when found, 0 when not found.
"""
self.ui.debug('imap', 'savemessage_fetchheaders called for %s: %s' % \
(headername, headervalue))
# run "fetch X:* rfc822.header"
# since we stored the mail we are looking for just recently, it would
# not be optimal to fetch all messages. So we'll find highest message
# UID in our local messagelist and search from there (exactly from
# UID+1). That works because UIDs are guaranteed to be unique and
# ascending.
if self.getmessagelist():
start = 1+max(self.getmessagelist().keys())
else:
# Folder was empty - start from 1
start = 1
# Imaplib quotes all parameters of a string type. That must not happen
# with the range X:*. So we use bytearray to stop imaplib from getting
# in our way
result = imapobj.uid('FETCH', bytearray('%d:*' % start), 'rfc822.header')
if result[0] != 'OK':
raise OfflineImapError('Error fetching mail headers: ' + '. '.join(result[1]),
OfflineImapError.ERROR.MESSAGE)
result = result[1]
found = 0
for item in result:
if found == 0 and type(item) == type( () ):
# Walk just tuples
if re.search("(?:^|\\r|\\n)%s:\s*%s(?:\\r|\\n)" % (headername, headervalue),
item[1], flags=re.IGNORECASE):
found = 1
elif found == 1:
if type(item) == type (""):
uid = re.search("UID\s+(\d+)", item, flags=re.IGNORECASE)
if uid:
return int(uid.group(1))
else:
self.ui.warn("Can't parse FETCH response, can't find UID: %s", result.__repr__())
else:
self.ui.warn("Can't parse FETCH response, we awaited string: %s", result.__repr__())
return 0
def getmessageinternaldate(self, content, rtime=None): def getmessageinternaldate(self, content, rtime=None):
"""Parses mail and returns an INTERNALDATE string """Parses mail and returns an INTERNALDATE string
@ -420,46 +495,64 @@ class IMAPFolder(BaseFolder):
self.savemessageflags(uid, flags) self.savemessageflags(uid, flags)
return uid return uid
try:
imapobj = self.imapserver.acquireconnection() imapobj = self.imapserver.acquireconnection()
try:
success = False # succeeded in APPENDING?
while not success:
# UIDPLUS extension provides us with an APPENDUID response.
use_uidplus = 'UIDPLUS' in imapobj.capabilities
# get the date of the message, so we can pass it to the server.
date = self.getmessageinternaldate(content, rtime)
content = re.sub("(?<!\r)\n", "\r\n", content)
if not use_uidplus:
# insert a random unique header that we can fetch later
(headername, headervalue) = self.generate_randomheader(
content)
self.ui.debug('imap', 'savemessage: header is: %s: %s' %\
(headername, headervalue))
content = self.savemessage_addheader(content, headername,
headervalue)
if len(content)>200:
dbg_output = "%s...%s" % (content[:150], content[-50:])
else:
dbg_output = content
self.ui.debug('imap', "savemessage: date: %s, content: '%s'" %
(date, dbg_output))
try: try:
imapobj.select(self.getfullname()) # Needed for search and making the box READ-WRITE # Select folder for append and make the box READ-WRITE
imapobj.select(self.getfullname())
except imapobj.readonly: except imapobj.readonly:
# readonly exception. Return original uid to notify that # readonly exception. Return original uid to notify that
# we did not save the message. (see savemessage in Base.py) # we did not save the message. (see savemessage in Base.py)
self.ui.msgtoreadonly(self, uid, content, flags) self.ui.msgtoreadonly(self, uid, content, flags)
return uid return uid
# UIDPLUS extension provides us with an APPENDUID response to our append() #Do the APPEND
use_uidplus = 'UIDPLUS' in imapobj.capabilities try:
(typ, dat) = imapobj.append(self.getfullname(),
# get the date of the message file, so we can pass it to the server.
date = self.getmessageinternaldate(content, rtime)
content = re.sub("(?<!\r)\n", "\r\n", content)
if not use_uidplus:
# insert a random unique header that we can fetch later
(headername, headervalue) = self.generate_randomheader(content)
self.ui.debug('imap', 'savemessage: new header is: %s: %s' % \
(headername, headervalue))
content = self.savemessage_addheader(content, headername,
headervalue)
if len(content)>200:
dbg_output = "%s...%s" % (content[:150],
content[-50:])
else:
dbg_output = content
self.ui.debug('imap', "savemessage: date: %s, content: '%s'" %
(date, dbg_output))
(typ,dat) = imapobj.append(self.getfullname(),
imaputil.flagsmaildir2imap(flags), imaputil.flagsmaildir2imap(flags),
date, content) date, content)
assert(typ == 'OK') success = True
except imapobj.abort, e:
# Checkpoint. Let it write out the messages, etc. # connection has been reset, release connection and retry.
self.ui.error(e, exc_info()[2])
self.imapserver.releaseconnection(imapobj, True)
imapobj = self.imapserver.acquireconnection()
except imapobj.error, e:
# If the server responds with 'BAD', append() raise()s directly.
# So we need to prepare a response ourselves.
typ, dat = 'BAD', str(e)
if typ != 'OK': #APPEND failed
raise OfflineImapError("Saving msg in folder '%s', repository "
"'%s' failed. Server reponded; %s %s\nMessage content was:"
" %s" % (self, self.getrepository(), typ, dat, dbg_output),
OfflineImapError.ERROR.MESSAGE)
# Checkpoint. Let it write out stuff, etc. Eg searches for
# just uploaded messages won't work if we don't do this.
(typ,dat) = imapobj.check() (typ,dat) = imapobj.check()
assert(typ == 'OK') assert(typ == 'OK')
@ -481,10 +574,15 @@ class IMAPFolder(BaseFolder):
headervalue) headervalue)
# See docs for savemessage in Base.py for explanation of this and other return values # See docs for savemessage in Base.py for explanation of this and other return values
if uid == 0: if uid == 0:
self.ui.debug('imap', 'savemessage: first attempt to get new UID failed. Going to run a NOOP and try again.') self.ui.debug('imap', 'savemessage: first attempt to get new UID failed. \
Going to run a NOOP and try again.')
assert(imapobj.noop()[0] == 'OK') assert(imapobj.noop()[0] == 'OK')
uid = self.savemessage_searchforheader(imapobj, headername, uid = self.savemessage_searchforheader(imapobj, headername,
headervalue) headervalue)
if uid == 0:
self.ui.debug('imap', 'savemessage: second attempt to get new UID failed. \
Going to try search headers manually')
uid = self.savemessage_fetchheaders(imapobj, headername, headervalue)
finally: finally:
self.imapserver.releaseconnection(imapobj) self.imapserver.releaseconnection(imapobj)
@ -549,7 +647,7 @@ class IMAPFolder(BaseFolder):
self.ui.flagstoreadonly(self, uidlist, flags) self.ui.flagstoreadonly(self, uidlist, flags)
return return
r = imapobj.uid('store', r = imapobj.uid('store',
imaputil.listjoin(uidlist), imaputil.uid_sequence(uidlist),
operation + 'FLAGS', operation + 'FLAGS',
imaputil.flagsmaildir2imap(flags)) imaputil.flagsmaildir2imap(flags))
assert r[0] == 'OK', 'Error with store: ' + '. '.join(r[1]) assert r[0] == 'OK', 'Error with store: ' + '. '.join(r[1])
@ -559,7 +657,7 @@ class IMAPFolder(BaseFolder):
# Some IMAP servers do not always return a result. Therefore, # Some IMAP servers do not always return a result. Therefore,
# only update the ones that it talks about, and manually fix # only update the ones that it talks about, and manually fix
# the others. # the others.
needupdate = copy(uidlist) needupdate = list(uidlist)
for result in r: for result in r:
if result == None: if result == None:
# Compensate for servers that don't return anything from # Compensate for servers that don't return anything from
@ -569,23 +667,18 @@ class IMAPFolder(BaseFolder):
if not ('UID' in attributehash and 'FLAGS' in attributehash): if not ('UID' in attributehash and 'FLAGS' in attributehash):
# Compensate for servers that don't return a UID attribute. # Compensate for servers that don't return a UID attribute.
continue continue
lflags = attributehash['FLAGS'] flagstr = attributehash['FLAGS']
uid = long(attributehash['UID']) uid = long(attributehash['UID'])
self.messagelist[uid]['flags'] = imaputil.flagsimap2maildir(lflags) self.messagelist[uid]['flags'] = imaputil.flagsimap2maildir(flagstr)
try: try:
needupdate.remove(uid) needupdate.remove(uid)
except ValueError: # Let it slide if it's not in the list except ValueError: # Let it slide if it's not in the list
pass pass
for uid in needupdate: for uid in needupdate:
if operation == '+': if operation == '+':
for flag in flags: self.messagelist[uid]['flags'] |= flags
if not flag in self.messagelist[uid]['flags']:
self.messagelist[uid]['flags'].append(flag)
self.messagelist[uid]['flags'].sort()
elif operation == '-': elif operation == '-':
for flag in flags: self.messagelist[uid]['flags'] -= flags
if flag in self.messagelist[uid]['flags']:
self.messagelist[uid]['flags'].remove(flag)
def deletemessage(self, uid): def deletemessage(self, uid):
self.deletemessages_noconvert([uid]) self.deletemessages_noconvert([uid])
@ -599,7 +692,7 @@ class IMAPFolder(BaseFolder):
if not len(uidlist): if not len(uidlist):
return return
self.addmessagesflags_noconvert(uidlist, ['T']) self.addmessagesflags_noconvert(uidlist, set('T'))
imapobj = self.imapserver.acquireconnection() imapobj = self.imapserver.acquireconnection()
try: try:
try: try:

View File

@ -1,6 +1,5 @@
# Local status cache virtual folder # Local status cache virtual folder
# Copyright (C) 2002 - 2008 John Goerzen # Copyright (C) 2002 - 2011 John Goerzen & contributors
# <jgoerzen@complete.org>
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by # it under the terms of the GNU General Public License as published by
@ -19,6 +18,10 @@
from Base import BaseFolder from Base import BaseFolder
import os import os
import threading import threading
try: # python 2.6 has set() built in
set
except NameError:
from sets import Set as set
magicline = "OFFLINEIMAP LocalStatus CACHE DATA - DO NOT MODIFY - FORMAT 1" magicline = "OFFLINEIMAP LocalStatus CACHE DATA - DO NOT MODIFY - FORMAT 1"
@ -28,7 +31,7 @@ class LocalStatusFolder(BaseFolder):
self.root = root self.root = root
self.sep = '.' self.sep = '.'
self.config = config self.config = config
self.filename = repository.getfolderfilename(name) self.filename = os.path.join(root, self.getfolderbasename())
self.messagelist = {} self.messagelist = {}
self.repository = repository self.repository = repository
self.savelock = threading.Lock() self.savelock = threading.Lock()
@ -80,11 +83,12 @@ class LocalStatusFolder(BaseFolder):
try: try:
uid, flags = line.split(':') uid, flags = line.split(':')
uid = long(uid) uid = long(uid)
flags = set(flags)
except ValueError, e: except ValueError, e:
errstr = "Corrupt line '%s' in cache file '%s'" % (line, self.filename) errstr = "Corrupt line '%s' in cache file '%s'" % \
(line, self.filename)
self.ui.warn(errstr) self.ui.warn(errstr)
raise ValueError(errstr) raise ValueError(errstr)
flags = [x for x in flags]
self.messagelist[uid] = {'uid': uid, 'flags': flags} self.messagelist[uid] = {'uid': uid, 'flags': flags}
file.close() file.close()
@ -95,8 +99,7 @@ class LocalStatusFolder(BaseFolder):
file.write(magicline + "\n") file.write(magicline + "\n")
for msg in self.messagelist.values(): for msg in self.messagelist.values():
flags = msg['flags'] flags = msg['flags']
flags.sort() flags = ''.join(sorted(flags))
flags = ''.join(flags)
file.write("%s:%s\n" % (msg['uid'], flags)) file.write("%s:%s\n" % (msg['uid'], flags))
file.flush() file.flush()
if self.doautosave: if self.doautosave:

View File

@ -23,6 +23,11 @@ try:
except: except:
pass #fail only if needed later on, not on import pass #fail only if needed later on, not on import
try: # python 2.6 has set() built in
set
except NameError:
from sets import Set as set
class LocalStatusSQLiteFolder(LocalStatusFolder): class LocalStatusSQLiteFolder(LocalStatusFolder):
"""LocalStatus backend implemented with an SQLite database """LocalStatus backend implemented with an SQLite database
@ -106,7 +111,8 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
if hasattr(self, 'connection'): if hasattr(self, 'connection'):
self.connection.close() #close old connections first self.connection.close() #close old connections first
self.connection = sqlite.connect(self.filename, check_same_thread = False) self.connection = sqlite.connect(self.filename,
check_same_thread = False)
if from_ver == 0: if from_ver == 0:
# from_ver==0: no db existent: plain text migration? # from_ver==0: no db existent: plain text migration?
@ -115,7 +121,7 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
plaintextfilename = os.path.join( plaintextfilename = os.path.join(
self.repository.account.getaccountmeta(), self.repository.account.getaccountmeta(),
'LocalStatus', 'LocalStatus',
re.sub('(^|\/)\.$','\\1dot', self.name)) self.getfolderbasename())
# MIGRATE from plaintext if needed # MIGRATE from plaintext if needed
if os.path.exists(plaintextfilename): if os.path.exists(plaintextfilename):
self.ui._msg('Migrating LocalStatus cache from plain text ' self.ui._msg('Migrating LocalStatus cache from plain text '
@ -127,7 +133,6 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
for line in file.xreadlines(): for line in file.xreadlines():
uid, flags = line.strip().split(':') uid, flags = line.strip().split(':')
uid = long(uid) uid = long(uid)
flags = list(flags)
flags = ''.join(sorted(flags)) flags = ''.join(sorted(flags))
data.append((uid,flags)) data.append((uid,flags))
self.connection.executemany('INSERT INTO status (id,flags) VALUES (?,?)', self.connection.executemany('INSERT INTO status (id,flags) VALUES (?,?)',
@ -167,7 +172,7 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
self.messagelist = {} self.messagelist = {}
cursor = self.connection.execute('SELECT id,flags from status') cursor = self.connection.execute('SELECT id,flags from status')
for row in cursor: for row in cursor:
flags = [x for x in row[1]] flags = set(row[1])
self.messagelist[row[0]] = {'uid': row[0], 'flags': flags} self.messagelist[row[0]] = {'uid': row[0], 'flags': flags}
def save(self): def save(self):
@ -227,8 +232,7 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
def savemessageflags(self, uid, flags): def savemessageflags(self, uid, flags):
self.messagelist[uid] = {'uid': uid, 'flags': flags} self.messagelist[uid] = {'uid': uid, 'flags': flags}
flags.sort() flags = ''.join(sorted(flags))
flags = ''.join(flags)
self.sql_write('UPDATE status SET flags=? WHERE id=?',(flags,uid)) self.sql_write('UPDATE status SET flags=? WHERE id=?',(flags,uid))
def deletemessages(self, uidlist): def deletemessages(self, uidlist):

View File

@ -28,6 +28,11 @@ try:
except ImportError: except ImportError:
from md5 import md5 from md5 import md5
try: # python 2.6 has set() built in
set
except NameError:
from sets import Set as set
from offlineimap import OfflineImapError from offlineimap import OfflineImapError
uidmatchre = re.compile(',U=(\d+)') uidmatchre = re.compile(',U=(\d+)')
@ -128,7 +133,7 @@ class MaildirFolder(BaseFolder):
folderstr = ',FMD5=' + foldermd5 folderstr = ',FMD5=' + foldermd5
for dirannex in ['new', 'cur']: for dirannex in ['new', 'cur']:
fulldirname = os.path.join(self.getfullname(), dirannex) fulldirname = os.path.join(self.getfullname(), dirannex)
files.extend(os.path.join(fulldirname, filename) for files.extend(os.path.join(dirannex, filename) for
filename in os.listdir(fulldirname)) filename in os.listdir(fulldirname))
for file in files: for file in files:
messagename = os.path.basename(file) messagename = os.path.basename(file)
@ -146,11 +151,10 @@ class MaildirFolder(BaseFolder):
#Check and see if the message is too big if the maxsize for this account is set #Check and see if the message is too big if the maxsize for this account is set
if(maxsize != -1): if(maxsize != -1):
filesize = os.path.getsize(file) size = os.path.getsize(os.path.join(self.getfullname(), file))
if(filesize > maxsize): if(size > maxsize):
continue continue
foldermatch = messagename.find(folderstr) != -1 foldermatch = messagename.find(folderstr) != -1
if not foldermatch: if not foldermatch:
# If there is no folder MD5 specified, or if it mismatches, # If there is no folder MD5 specified, or if it mismatches,
@ -166,11 +170,13 @@ class MaildirFolder(BaseFolder):
nouidcounter -= 1 nouidcounter -= 1
else: else:
uid = long(uidmatch.group(1)) uid = long(uidmatch.group(1))
#identify flags in the path name
flagmatch = self.flagmatchre.search(messagename) flagmatch = self.flagmatchre.search(messagename)
flags = []
if flagmatch: if flagmatch:
flags = [x for x in flagmatch.group(1)] flags = set(flagmatch.group(1))
flags.sort() else:
flags = set()
# 'filename' is 'dirannex/filename', e.g. cur/123_U=1_FMD5=1:2,S
retval[uid] = {'uid': uid, retval[uid] = {'uid': uid,
'flags': flags, 'flags': flags,
'filename': file} 'filename': file}
@ -261,7 +267,7 @@ class MaildirFolder(BaseFolder):
if rtime != None: if rtime != None:
os.utime(os.path.join(tmpdir, messagename), (rtime, rtime)) os.utime(os.path.join(tmpdir, messagename), (rtime, rtime))
self.messagelist[uid] = {'uid': uid, 'flags': [], self.messagelist[uid] = {'uid': uid, 'flags': set(),
'filename': os.path.join('tmp', messagename)} 'filename': os.path.join('tmp', messagename)}
# savemessageflags moves msg to 'cur' or 'new' as appropriate # savemessageflags moves msg to 'cur' or 'new' as appropriate
self.savemessageflags(uid, flags) self.savemessageflags(uid, flags)
@ -288,14 +294,19 @@ class MaildirFolder(BaseFolder):
infostr = infomatch.group(1) infostr = infomatch.group(1)
newname = newname.split(self.infosep)[0] # Strip off the info string. newname = newname.split(self.infosep)[0] # Strip off the info string.
infostr = re.sub('2,[A-Z]*', '', infostr) infostr = re.sub('2,[A-Z]*', '', infostr)
flags.sort() infostr += '2,' + ''.join(sorted(flags))
infostr += '2,' + ''.join(flags)
newname += infostr newname += infostr
newfilename = os.path.join(dir_prefix, newname) newfilename = os.path.join(dir_prefix, newname)
if (newfilename != oldfilename): if (newfilename != oldfilename):
try:
os.rename(os.path.join(self.getfullname(), oldfilename), os.rename(os.path.join(self.getfullname(), oldfilename),
os.path.join(self.getfullname(), newfilename)) os.path.join(self.getfullname(), newfilename))
except OSError, e:
raise OfflineImapError("Can't rename file '%s' to '%s': %s" % (
oldfilename, newfilename, e[1]),
OfflineImapError.ERROR.FOLDER)
self.messagelist[uid]['flags'] = flags self.messagelist[uid]['flags'] = flags
self.messagelist[uid]['filename'] = newfilename self.messagelist[uid]['filename'] = newfilename

View File

@ -2,7 +2,7 @@
"""Threaded IMAP4 client. """Threaded IMAP4 client.
Based on RFC 2060 and original imaplib module. Based on RFC 3501 and original imaplib module.
Public classes: IMAP4 Public classes: IMAP4
IMAP4_SSL IMAP4_SSL
@ -17,9 +17,9 @@ Public functions: Internaldate2Time
__all__ = ("IMAP4", "IMAP4_SSL", "IMAP4_stream", __all__ = ("IMAP4", "IMAP4_SSL", "IMAP4_stream",
"Internaldate2Time", "ParseFlags", "Time2Internaldate") "Internaldate2Time", "ParseFlags", "Time2Internaldate")
__version__ = "2.24" __version__ = "2.28"
__release__ = "2" __release__ = "2"
__revision__ = "24" __revision__ = "28"
__credits__ = """ __credits__ = """
Authentication code contributed by Donn Cave <donn@u.washington.edu> June 1998. Authentication code contributed by Donn Cave <donn@u.washington.edu> June 1998.
String method conversion by ESR, February 2001. String method conversion by ESR, February 2001.
@ -38,7 +38,8 @@ Improved timeout handling contributed by Ivan Vovnenko <ivovnenko@gmail.com> Oct
Timeout handling further improved by Ethan Glasser-Camp <glasse@cs.rpi.edu> December 2010. Timeout handling further improved by Ethan Glasser-Camp <glasse@cs.rpi.edu> December 2010.
Time2Internaldate() patch to match RFC2060 specification of English month names from bugs.python.org/issue11024 March 2011. Time2Internaldate() patch to match RFC2060 specification of English month names from bugs.python.org/issue11024 March 2011.
starttls() bug fixed with the help of Sebastian Spaeth <sebastian@sspaeth.de> April 2011. starttls() bug fixed with the help of Sebastian Spaeth <sebastian@sspaeth.de> April 2011.
Threads now set the "daemon" flag (suggested by offlineimap-project).""" Threads now set the "daemon" flag (suggested by offlineimap-project) April 2011.
Single quoting introduced with the help of Vladimir Marek <vladimir.marek@oracle.com> August 2011."""
__author__ = "Piers Lauder <piers@janeelix.com>" __author__ = "Piers Lauder <piers@janeelix.com>"
__URL__ = "http://imaplib2.sourceforge.net" __URL__ = "http://imaplib2.sourceforge.net"
__license__ = "Python License" __license__ = "Python License"
@ -88,7 +89,7 @@ Commands = {
'GETANNOTATION':((AUTH, SELECTED), True), 'GETANNOTATION':((AUTH, SELECTED), True),
'GETQUOTA': ((AUTH, SELECTED), True), 'GETQUOTA': ((AUTH, SELECTED), True),
'GETQUOTAROOT': ((AUTH, SELECTED), True), 'GETQUOTAROOT': ((AUTH, SELECTED), True),
'ID': ((NONAUTH, AUTH, SELECTED), True), 'ID': ((NONAUTH, AUTH, LOGOUT, SELECTED), True),
'IDLE': ((SELECTED,), False), 'IDLE': ((SELECTED,), False),
'LIST': ((AUTH, SELECTED), True), 'LIST': ((AUTH, SELECTED), True),
'LOGIN': ((NONAUTH,), False), 'LOGIN': ((NONAUTH,), False),
@ -137,11 +138,14 @@ class Request(object):
"""Private class to represent a request awaiting response.""" """Private class to represent a request awaiting response."""
def __init__(self, parent, name=None, callback=None, cb_arg=None): def __init__(self, parent, name=None, callback=None, cb_arg=None, cb_self=False):
self.parent = parent self.parent = parent
self.name = name self.name = name
self.callback = callback # Function called to process result self.callback = callback # Function called to process result
if not cb_self:
self.callback_arg = cb_arg # Optional arg passed to "callback" self.callback_arg = cb_arg # Optional arg passed to "callback"
else:
self.callback_arg = (self, cb_arg) # Self reference required in callback arg
self.tag = '%s%s' % (parent.tagpre, parent.tagnum) self.tag = '%s%s' % (parent.tagpre, parent.tagnum)
parent.tagnum += 1 parent.tagnum += 1
@ -153,9 +157,6 @@ class Request(object):
def abort(self, typ, val): def abort(self, typ, val):
"""Called whenever we abort a command
Sets self.aborted reason, and deliver()s nothing"""
self.aborted = (typ, val) self.aborted = (typ, val)
self.deliver(None) self.deliver(None)
@ -238,12 +239,17 @@ class IMAP4(object):
All (non-callback) arguments to commands are converted to strings, All (non-callback) arguments to commands are converted to strings,
except for AUTHENTICATE, and the last argument to APPEND which is except for AUTHENTICATE, and the last argument to APPEND which is
passed as an IMAP4 literal. If necessary (the string contains any passed as an IMAP4 literal. If necessary (the string contains any
non-printing characters or white-space and isn't enclosed with either non-printing characters or white-space and isn't enclosed with
parentheses or double quotes) each string is quoted. However, the either parentheses or double or single quotes) each string is
'password' argument to the LOGIN command is always quoted. If you quoted. However, the 'password' argument to the LOGIN command is
want to avoid having an argument string quoted (eg: the 'flags' always quoted. If you want to avoid having an argument string
argument to STORE) then enclose the string in parentheses (eg: quoted (eg: the 'flags' argument to STORE) then enclose the string
"(\Deleted)"). in parentheses (eg: "(\Deleted)"). If you are using "sequence sets"
containing the wildcard character '*', then enclose the argument
in single quotes: the quotes will be removed and the resulting
string passed unquoted. Note also that you can pass in an argument
with a type that doesn't evaluate to 'basestring' (eg: 'bytearray')
and it will be converted to a string without quoting.
There is one instance variable, 'state', that is useful for tracking There is one instance variable, 'state', that is useful for tracking
whether the client needs to login to the server. If it has the whether the client needs to login to the server. If it has the
@ -275,6 +281,7 @@ class IMAP4(object):
# so match not the inverse set # so match not the inverse set
mustquote_cre = re.compile(r"[^!#$&'+,./0-9:;<=>?@A-Z\[^_`a-z|}~-]") mustquote_cre = re.compile(r"[^!#$&'+,./0-9:;<=>?@A-Z\[^_`a-z|}~-]")
response_code_cre = re.compile(r'\[(?P<type>[A-Z-]+)( (?P<data>[^\]]*))?\]') response_code_cre = re.compile(r'\[(?P<type>[A-Z-]+)( (?P<data>[^\]]*))?\]')
# sequence_set_cre = re.compile(r"^[0-9]+(:([0-9]+|\*))?(,[0-9]+(:([0-9]+|\*))?)*$")
untagged_response_cre = re.compile(r'\* (?P<type>[A-Z-]+)( (?P<data>.*))?') untagged_response_cre = re.compile(r'\* (?P<type>[A-Z-]+)( (?P<data>.*))?')
untagged_status_cre = re.compile(r'\* (?P<data>\d+) (?P<type>[A-Z-]+)( (?P<data2>.*))?') untagged_status_cre = re.compile(r'\* (?P<data>\d+) (?P<type>[A-Z-]+)( (?P<data2>.*))?')
@ -339,8 +346,6 @@ class IMAP4(object):
self.state_change_free = threading.Event() self.state_change_free = threading.Event()
self.state_change_pending = threading.Lock() self.state_change_pending = threading.Lock()
self.commands_lock = threading.Lock() self.commands_lock = threading.Lock()
"""commands_lock prevents self.untagged_responses to be
manipulated concurrently"""
self.idle_lock = threading.Lock() self.idle_lock = threading.Lock()
self.ouq = Queue.Queue(10) self.ouq = Queue.Queue(10)
@ -368,7 +373,7 @@ class IMAP4(object):
elif self._get_untagged_response('OK'): elif self._get_untagged_response('OK'):
if __debug__: self._log(1, 'state => NONAUTH') if __debug__: self._log(1, 'state => NONAUTH')
else: else:
raise self.error(self.welcome) raise self.error('unrecognised server welcome message: %s' % `self.welcome`)
typ, dat = self.capability() typ, dat = self.capability()
if dat == [None]: if dat == [None]:
@ -443,6 +448,35 @@ class IMAP4(object):
return s return s
def ssl_wrap_socket(self):
# Allow sending of keep-alive messages - seems to prevent some servers
# from closing SSL, leading to deadlocks.
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
try:
import ssl
if self.ca_certs is not None:
cert_reqs = ssl.CERT_REQUIRED
else:
cert_reqs = ssl.CERT_NONE
self.sock = ssl.wrap_socket(self.sock, self.keyfile, self.certfile, ca_certs=self.ca_certs, cert_reqs=cert_reqs)
ssl_exc = ssl.SSLError
except ImportError:
# No ssl module, and socket.ssl does not allow certificate verification
if self.ca_certs is not None:
raise socket.sslerror("SSL CA certificates cannot be checked without ssl module")
self.sock = socket.ssl(self.sock, self.keyfile, self.certfile)
ssl_exc = socket.sslerror
if self.cert_verify_cb is not None:
cert_err = self.cert_verify_cb(self.sock.getpeercert(), self.host)
if cert_err:
raise ssl_exc(cert_err)
self.read_fd = self.sock.fileno()
def start_compressing(self): def start_compressing(self):
"""start_compressing() """start_compressing()
Enable deflate compression on the socket (RFC 4978).""" Enable deflate compression on the socket (RFC 4978)."""
@ -671,7 +705,7 @@ class IMAP4(object):
def examine(self, mailbox='INBOX', **kw): def examine(self, mailbox='INBOX', **kw):
"""(typ, [data]) = examine(mailbox='INBOX', readonly=False) """(typ, [data]) = examine(mailbox='INBOX')
Select a mailbox for READ-ONLY access. (Flushes all untagged responses.) Select a mailbox for READ-ONLY access. (Flushes all untagged responses.)
'data' is count of messages in mailbox ('EXISTS' response). 'data' is count of messages in mailbox ('EXISTS' response).
Mandated responses are ('FLAGS', 'EXISTS', 'RECENT', 'UIDVALIDITY'), so Mandated responses are ('FLAGS', 'EXISTS', 'RECENT', 'UIDVALIDITY'), so
@ -745,13 +779,23 @@ class IMAP4(object):
def id(self, *kv_pairs, **kw): def id(self, *kv_pairs, **kw):
"""(typ, [data]) = <instance>.id(kv_pairs) """(typ, [data]) = <instance>.id(kv_pairs)
'data' is list of ID key value pairs. 'kv_pairs' is a possibly empty list of keys and values.
Request information for problem analysis and determination. 'data' is a list of ID key value pairs or NIL.
NB: a single argument is assumed to be correctly formatted and is passed through unchanged
(for backward compatibility with earlier version).
Exchange information for problem analysis and determination.
The ID extension is defined in RFC 2971. """ The ID extension is defined in RFC 2971. """
name = 'ID' name = 'ID'
kw['untagged_response'] = name kw['untagged_response'] = name
return self._simple_command(name, *kv_pairs, **kw)
if not kv_pairs:
data = 'NIL'
elif len(kv_pairs) == 1:
data = kv_pairs[0] # Assume invoker passing correctly formatted string (back-compat)
else:
data = '(%s)' % ' '.join([(arg and self._quote(arg) or 'NIL') for arg in kv_pairs])
return self._simple_command(name, (data,), **kw)
def idle(self, timeout=None, **kw): def idle(self, timeout=None, **kw):
@ -999,8 +1043,8 @@ class IMAP4(object):
return self._simple_command(name, sort_criteria, charset, *search_criteria, **kw) return self._simple_command(name, sort_criteria, charset, *search_criteria, **kw)
def starttls(self, keyfile=None, certfile=None, **kw): def starttls(self, keyfile=None, certfile=None, ca_certs=None, cert_verify_cb=None, **kw):
"""(typ, [data]) = starttls(keyfile=None, certfile=None) """(typ, [data]) = starttls(keyfile=None, certfile=None, ca_certs=None, cert_verify_cb=None)
Start TLS negotiation as per RFC 2595.""" Start TLS negotiation as per RFC 2595."""
name = 'STARTTLS' name = 'STARTTLS'
@ -1031,14 +1075,13 @@ class IMAP4(object):
self.rdth.start() self.rdth.start()
raise self.error("Couldn't establish TLS session: %s" % dat) raise self.error("Couldn't establish TLS session: %s" % dat)
try: self.keyfile = keyfile
try: self.certfile = certfile
import ssl self.ca_certs = ca_certs
self.sock = ssl.wrap_socket(self.sock, keyfile, certfile) self.cert_verify_cb = cert_verify_cb
except ImportError:
self.sock = socket.ssl(self.sock, keyfile, certfile)
self.read_fd = self.sock.fileno() try:
self.ssl_wrap_socket()
finally: finally:
# Restart reader thread # Restart reader thread
self.rdth = threading.Thread(target=self._reader) self.rdth = threading.Thread(target=self._reader)
@ -1140,29 +1183,34 @@ class IMAP4(object):
def _append_untagged(self, typ, dat): def _append_untagged(self, typ, dat):
"""Append new untagged response
Append new 'dat' to end of last untagged response if same 'typ', # Append new 'dat' to end of last untagged response if same 'typ',
else append new response.""" # else append new response.
if dat is None: dat = '' if dat is None: dat = ''
ur_data = []
self.commands_lock.acquire() # protect untagged_responses self.commands_lock.acquire()
if self.untagged_responses and self.untagged_responses[-1][0] == typ: if self.untagged_responses:
# last respons is of type 'typ', get ur_data for appending urn, urd = self.untagged_responses[-1]
ur_data = self.untagged_responses[-1][1] if urn != typ:
urd = None
else: else:
# need to create new untagged response of this type urd = None
self.untagged_responses.append([typ, ur_data])
if urd is None:
urd = []
self.untagged_responses.append([typ, urd])
urd.append(dat)
ur_data.append(dat)
self.commands_lock.release() self.commands_lock.release()
if __debug__: self._log(5, 'untagged_responses[%s] %s += ["%s"]' % (typ, len(ur_data)-1, dat))
if __debug__: self._log(5, 'untagged_responses[%s] %s += ["%s"]' % (typ, len(urd)-1, dat))
def _check_bye(self): def _check_bye(self):
"""raise Exception if untagged responses contains a 'BYE'"""
bye = self._get_untagged_response('BYE', leave=True) bye = self._get_untagged_response('BYE', leave=True)
if bye: if bye:
raise self.abort(bye[-1]) raise self.abort(bye[-1])
@ -1171,12 +1219,14 @@ class IMAP4(object):
def _checkquote(self, arg): def _checkquote(self, arg):
# Must quote command args if "atom-specials" present, # Must quote command args if "atom-specials" present,
# and not already quoted. # and not already quoted. NB: single quotes are removed.
if not isinstance(arg, basestring): if not isinstance(arg, basestring):
return arg return arg
if len(arg) >= 2 and (arg[0],arg[-1]) in (('(',')'),('"','"')): if len(arg) >= 2 and (arg[0],arg[-1]) in (('(',')'),('"','"')):
return arg return arg
if len(arg) >= 2 and (arg[0],arg[-1]) in (("'","'"),):
return arg[1:-1]
if arg and self.mustquote_cre.search(arg) is None: if arg and self.mustquote_cre.search(arg) is None:
return arg return arg
return self._quote(arg) return self._quote(arg)
@ -1372,11 +1422,7 @@ class IMAP4(object):
def _get_untagged_response(self, name, leave=False): def _get_untagged_response(self, name, leave=False):
"""Return an untagged response of type 'name'
:param leave: If leave (default: False) is True, we keep the
fetched responsem; otherwise it will be deleted. Returns
None if no such response found."""
self.commands_lock.acquire() self.commands_lock.acquire()
for i, (typ, dat) in enumerate(self.untagged_responses): for i, (typ, dat) in enumerate(self.untagged_responses):
@ -1543,24 +1589,13 @@ class IMAP4(object):
def _simple_command(self, name, *args, **kw): def _simple_command(self, name, *args, **kw):
if 'callback' in kw: if 'callback' in kw:
rqb = self._command(name, callback=self._command_completer, *args) self._command(name, *args, callback=self._command_completer, cb_arg=kw, cb_self=True)
rqb.callback_arg = (rqb, kw)
return (None, None) return (None, None)
return self._command_complete(self._command(name, *args), kw) return self._command_complete(self._command(name, *args), kw)
def _untagged_response(self, typ, dat, name): def _untagged_response(self, typ, dat, name):
"""Returns an untagged response for 'name' of type 'typ'
:param typ: 'OK, 'NO', etc... which will be used for the type of
the response.
:param dat: The fallback data to be used in case `typ` is
'NO'. Otherwise the data from the existing untagged
responses will be searched for data to be returned. If there
is no such response, we return `[None]` as data.
:param name: The name of the response.
:returns: (typ, data)
"""
if typ == 'NO': if typ == 'NO':
return typ, dat return typ, dat
data = self._get_untagged_response(name) data = self._get_untagged_response(name)
@ -1940,18 +1975,23 @@ class IMAP4_SSL(IMAP4):
port - port number (default: standard IMAP4 SSL port); port - port number (default: standard IMAP4 SSL port);
keyfile - PEM formatted file that contains your private key (default: None); keyfile - PEM formatted file that contains your private key (default: None);
certfile - PEM formatted certificate chain file (default: None); certfile - PEM formatted certificate chain file (default: None);
ca_certs - PEM formatted certificate chain file used to validate server certificates (default: None);
cert_verify_cb - function to verify authenticity of server certificates (default: None);
debug - debug level (default: 0 - no debug); debug - debug level (default: 0 - no debug);
debug_file - debug stream (default: sys.stderr); debug_file - debug stream (default: sys.stderr);
identifier - thread identifier prefix (default: host); identifier - thread identifier prefix (default: host);
timeout - timeout in seconds when expecting a command response. timeout - timeout in seconds when expecting a command response.
debug_buf_lvl - debug level at which buffering is turned off.
For more documentation see the docstring of the parent class IMAP4. For more documentation see the docstring of the parent class IMAP4.
""" """
def __init__(self, host=None, port=None, keyfile=None, certfile=None, debug=None, debug_file=None, identifier=None, timeout=None, debug_buf_lvl=None): def __init__(self, host=None, port=None, keyfile=None, certfile=None, ca_certs=None, cert_verify_cb=None, debug=None, debug_file=None, identifier=None, timeout=None, debug_buf_lvl=None):
self.keyfile = keyfile self.keyfile = keyfile
self.certfile = certfile self.certfile = certfile
self.ca_certs = ca_certs
self.cert_verify_cb = cert_verify_cb
IMAP4.__init__(self, host, port, debug, debug_file, identifier, timeout, debug_buf_lvl) IMAP4.__init__(self, host, port, debug, debug_file, identifier, timeout, debug_buf_lvl)
@ -1965,14 +2005,7 @@ class IMAP4_SSL(IMAP4):
self.host = self._choose_nonull_or_dflt('', host) self.host = self._choose_nonull_or_dflt('', host)
self.port = self._choose_nonull_or_dflt(IMAP4_SSL_PORT, port) self.port = self._choose_nonull_or_dflt(IMAP4_SSL_PORT, port)
self.sock = self.open_socket() self.sock = self.open_socket()
self.ssl_wrap_socket()
try:
import ssl
self.sslobj = ssl.wrap_socket(self.sock, self.keyfile, self.certfile)
except ImportError:
self.sslobj = socket.ssl(self.sock, self.keyfile, self.certfile)
self.read_fd = self.sock.fileno()
def read(self, size): def read(self, size):
@ -1980,12 +2013,12 @@ class IMAP4_SSL(IMAP4):
Read at most 'size' bytes from remote.""" Read at most 'size' bytes from remote."""
if self.decompressor is None: if self.decompressor is None:
return self.sslobj.read(size) return self.sock.read(size)
if self.decompressor.unconsumed_tail: if self.decompressor.unconsumed_tail:
data = self.decompressor.unconsumed_tail data = self.decompressor.unconsumed_tail
else: else:
data = self.sslobj.read(8192) data = self.sock.read(8192)
return self.decompressor.decompress(data, size) return self.decompressor.decompress(data, size)
@ -1998,10 +2031,12 @@ class IMAP4_SSL(IMAP4):
data = self.compressor.compress(data) data = self.compressor.compress(data)
data += self.compressor.flush(zlib.Z_SYNC_FLUSH) data += self.compressor.flush(zlib.Z_SYNC_FLUSH)
# NB: socket.ssl needs a "sendall" method to match socket objects. if hasattr(self.sock, "sendall"):
self.sock.sendall(data)
else:
bytes = len(data) bytes = len(data)
while bytes > 0: while bytes > 0:
sent = self.sslobj.write(data) sent = self.sock.write(data)
if sent == bytes: if sent == bytes:
break # avoid copy break # avoid copy
data = data[sent:] data = data[sent:]
@ -2012,7 +2047,7 @@ class IMAP4_SSL(IMAP4):
"""ssl = ssl() """ssl = ssl()
Return socket.ssl instance used to communicate with the IMAP4 server.""" Return socket.ssl instance used to communicate with the IMAP4 server."""
return self.sslobj return self.sock
@ -2021,13 +2056,14 @@ class IMAP4_stream(IMAP4):
"""IMAP4 client class over a stream """IMAP4 client class over a stream
Instantiate with: Instantiate with:
IMAP4_stream(command, debug=None, debug_file=None, identifier=None, timeout=None) IMAP4_stream(command, debug=None, debug_file=None, identifier=None, timeout=None, debug_buf_lvl=None)
command - string that can be passed to subprocess.Popen(); command - string that can be passed to subprocess.Popen();
debug - debug level (default: 0 - no debug); debug - debug level (default: 0 - no debug);
debug_file - debug stream (default: sys.stderr); debug_file - debug stream (default: sys.stderr);
identifier - thread identifier prefix (default: host); identifier - thread identifier prefix (default: host);
timeout - timeout in seconds when expecting a command response. timeout - timeout in seconds when expecting a command response.
debug_buf_lvl - debug level at which buffering is turned off.
For more documentation see the docstring of the parent class IMAP4. For more documentation see the docstring of the parent class IMAP4.
""" """
@ -2296,7 +2332,7 @@ if __name__ == '__main__':
('list', ('/tmp', 'imaplib2_test*')), ('list', ('/tmp', 'imaplib2_test*')),
('select', ('/tmp/imaplib2_test.2',)), ('select', ('/tmp/imaplib2_test.2',)),
('search', (None, 'SUBJECT', 'IMAP4 test')), ('search', (None, 'SUBJECT', 'IMAP4 test')),
('fetch', ('1', '(FLAGS INTERNALDATE RFC822)')), ('fetch', ("'1:*'", '(FLAGS INTERNALDATE RFC822)')),
('store', ('1', 'FLAGS', '(\Deleted)')), ('store', ('1', 'FLAGS', '(\Deleted)')),
('namespace', ()), ('namespace', ()),
('expunge', ()), ('expunge', ()),
@ -2380,6 +2416,11 @@ if __name__ == '__main__':
else: path = ml.split()[-1] else: path = ml.split()[-1]
run('delete', (path,)) run('delete', (path,))
if 'ID' in M.capabilities:
run('id', ())
run('id', ('("name", "imaplib2")',))
run('id', ("version", __version__, "os", os.uname()[0]))
for cmd,args in test_seq2: for cmd,args in test_seq2:
if (cmd,args) != ('uid', ('SEARCH', 'SUBJECT', 'IMAP4 test')): if (cmd,args) != ('uid', ('SEARCH', 'SUBJECT', 'IMAP4 test')):
run(cmd, args) run(cmd, args)

View File

@ -21,8 +21,10 @@ import re
import socket import socket
import time import time
import subprocess import subprocess
from offlineimap.ui import getglobalui
import threading import threading
from hashlib import sha1
from offlineimap.ui import getglobalui
from offlineimap import OfflineImapError from offlineimap import OfflineImapError
from offlineimap.imaplib2 import IMAP4, IMAP4_SSL, zlib, IMAP4_PORT, InternalDate, Mon2num from offlineimap.imaplib2 import IMAP4, IMAP4_SSL, zlib, IMAP4_PORT, InternalDate, Mon2num
@ -49,7 +51,14 @@ class UsefulIMAPMixIn:
return return
# Wipe out all old responses, to maintain semantics with old imaplib2 # Wipe out all old responses, to maintain semantics with old imaplib2
del self.untagged_responses[:] del self.untagged_responses[:]
try:
result = self.__class__.__bases__[1].select(self, mailbox, readonly) result = self.__class__.__bases__[1].select(self, mailbox, readonly)
except self.abort, e:
# self.abort is raised when we are supposed to retry
errstr = "Server '%s' closed connection, error on SELECT '%s'. Ser"\
"ver said: %s" % (self.host, mailbox, e.args[0])
severity = OfflineImapError.ERROR.FOLDER_RETRY
raise OfflineImapError(errstr, severity)
if result[0] != 'OK': if result[0] != 'OK':
#in case of error, bail out with OfflineImapError #in case of error, bail out with OfflineImapError
errstr = "Error SELECTing mailbox '%s', server reply:\n%s" %\ errstr = "Error SELECTing mailbox '%s', server reply:\n%s" %\
@ -127,164 +136,33 @@ def new_mesg(self, s, tn=None, secs=None):
tm = time.strftime('%M:%S', time.localtime(secs)) tm = time.strftime('%M:%S', time.localtime(secs))
getglobalui().debug('imap', ' %s.%02d %s %s' % (tm, (secs*100)%100, tn, s)) getglobalui().debug('imap', ' %s.%02d %s %s' % (tm, (secs*100)%100, tn, s))
class WrappedIMAP4_SSL(UsefulIMAPMixIn, IMAP4_SSL):
"""Provides an improved version of the standard IMAP4_SSL
It provides a better readline() implementation as impaplib's class WrappedIMAP4_SSL(UsefulIMAPMixIn, IMAP4_SSL):
readline() is extremly inefficient. It can also connect to IPv6 """Improved version of imaplib.IMAP4_SSL overriding select()"""
addresses."""
def __init__(self, *args, **kwargs): def __init__(self, *args, **kwargs):
self._readbuf = '' self._fingerprint = kwargs.get('fingerprint', None)
self._cacertfile = kwargs.get('cacertfile', None) if kwargs.has_key('fingerprint'):
if kwargs.has_key('cacertfile'): del kwargs['fingerprint']
del kwargs['cacertfile'] super(WrappedIMAP4_SSL, self).__init__(*args, **kwargs)
IMAP4_SSL.__init__(self, *args, **kwargs)
def open(self, host=None, port=None): def open(self, host=None, port=None):
"""Do whatever IMAP4_SSL would do in open, but call sslwrap super(WrappedIMAP4_SSL, self).open(host, port)
with cert verification""" if (self._fingerprint or not self.ca_certs) and\
#IMAP4_SSL.open(self, host, port) uses the below 2 lines: 'ssl' in locals(): # <--disable for python 2.5
self.host = host # compare fingerprints
self.port = port fingerprint = sha1(self.sock.getpeercert(True)).hexdigest()
if fingerprint != self._fingerprint:
raise OfflineImapError("Server SSL fingerprint '%s' for hostnam"
"e '%s' does not match configured fingerprint. Please ver"
"ify and set 'cert_fingerprint' accordingly if not set ye"
"t." % (fingerprint, host),
OfflineImapError.ERROR.REPO)
#rather than just self.sock = socket.create_connection((host, port))
#we use the below part to be able to connect to ipv6 addresses too
#This connects to the first ip found ipv4/ipv6
#Added by Adriaan Peeters <apeeters@lashout.net> based on a socket
#example from the python documentation:
#http://www.python.org/doc/lib/socket-example.html
res = socket.getaddrinfo(host, port, socket.AF_UNSPEC,
socket.SOCK_STREAM)
# Try all the addresses in turn until we connect()
last_error = 0
for remote in res:
af, socktype, proto, canonname, sa = remote
self.sock = socket.socket(af, socktype, proto)
last_error = self.sock.connect_ex(sa)
if last_error == 0:
break
else:
self.sock.close()
if last_error != 0:
raise Exception("can't open socket; error: %s"\
% socket.error(last_error))
# Allow sending of keep-alive message seems to prevent some servers
# from closing SSL on us leading to deadlocks
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
#connected to socket, now wrap it in SSL
try:
if self._cacertfile:
requirecert = ssl.CERT_REQUIRED
else:
requirecert = ssl.CERT_NONE
self.sslobj = ssl.wrap_socket(self.sock, self.keyfile,
self.certfile,
ca_certs = self._cacertfile,
cert_reqs = requirecert)
except NameError:
#Python 2.4/2.5 don't have the ssl module, we need to
#socket.ssl() here but that doesn't allow cert
#verification!!!
if self._cacertfile:
#user configured a CA certificate, but python 2.4/5 doesn't
#allow us to easily check it. So bail out here.
raise Exception("SSL CA Certificates cannot be checked with python <=2.6. Abort")
self.sslobj = socket.ssl(self.sock, self.keyfile,
self.certfile)
else:
#ssl.wrap_socket worked and cert is verified (if configured),
#now check that hostnames also match if we have a CA cert.
if self._cacertfile:
error = self._verifycert(self.sslobj.getpeercert(), host)
if error:
raise ssl.SSLError("SSL Certificate host name mismatch: %s" % error)
# imaplib2 uses this to poll()
self.read_fd = self.sock.fileno()
#TODO: Done for now. We should implement a mutt-like behavior
#that offers the users to accept a certificate (presenting a
#fingerprint of it) (get via self.sslobj.getpeercert()), and
#save that, and compare on future connects, rather than having
#to trust what the CA certs say.
def _verifycert(self, cert, hostname):
'''Verify that cert (in socket.getpeercert() format) matches hostname.
CRLs are not handled.
Returns error message if any problems are found and None on success.
'''
if not cert:
return ('no certificate received')
dnsname = hostname.lower()
certnames = []
# cert expired?
notafter = cert.get('notAfter')
if notafter:
if time.time() >= ssl.cert_time_to_seconds(notafter):
return ('server certificate error: certificate expired %s'
) % notafter
# First read commonName
for s in cert.get('subject', []):
key, value = s[0]
if key == 'commonName':
certnames.append(value.lower())
if len(certnames) == 0:
return ('no commonName found in certificate')
# Then read subjectAltName
for key, value in cert.get('subjectAltName', []):
if key == 'DNS':
certnames.append(value.lower())
# And finally try to match hostname with one of these names
for certname in certnames:
if (certname == dnsname or
'.' in dnsname and certname == '*.' + dnsname.split('.', 1)[1]):
return None
return ('no matching domain name found in certificate')
class WrappedIMAP4(UsefulIMAPMixIn, IMAP4): class WrappedIMAP4(UsefulIMAPMixIn, IMAP4):
"""Improved version of imaplib.IMAP4 that can also connect to IPv6""" """Improved version of imaplib.IMAP4 overriding select()"""
pass
def open(self, host = '', port = IMAP4_PORT):
"""Setup connection to remote server on "host:port"
(default: localhost:standard IMAP4 port).
"""
#self.host and self.port are needed by the parent IMAP4 class
self.host = host
self.port = port
res = socket.getaddrinfo(host, port, socket.AF_UNSPEC,
socket.SOCK_STREAM)
# Try each address returned by getaddrinfo in turn until we
# manage to connect to one.
# Try all the addresses in turn until we connect()
last_error = 0
for remote in res:
af, socktype, proto, canonname, sa = remote
self.sock = socket.socket(af, socktype, proto)
last_error = self.sock.connect_ex(sa)
if last_error == 0:
break
else:
self.sock.close()
if last_error != 0:
raise Exception("can't open socket; error: %s"\
% socket.error(last_error))
self.file = self.sock.makefile('rb')
# imaplib2 uses this to poll()
self.read_fd = self.sock.fileno()
mustquote = re.compile(r"[^\w!#$%&'+,.:;<=>?^`|~-]")
def Internaldate2epoch(resp): def Internaldate2epoch(resp):
"""Convert IMAP4 INTERNALDATE to UT. """Convert IMAP4 INTERNALDATE to UT.

View File

@ -24,10 +24,12 @@ import offlineimap.accounts
import hmac import hmac
import socket import socket
import base64 import base64
import time
import errno
from sys import exc_info
from socket import gaierror from socket import gaierror
try: try:
from ssl import SSLError from ssl import SSLError, cert_time_to_seconds
except ImportError: except ImportError:
# Protect against python<2.6, use dummy and won't get SSL errors. # Protect against python<2.6, use dummy and won't get SSL errors.
SSLError = None SSLError = None
@ -42,58 +44,58 @@ except ImportError:
pass pass
class IMAPServer: class IMAPServer:
"""Initializes all variables from an IMAPRepository() instance
Various functions, such as acquireconnection() return an IMAP4
object on which we can operate."""
GSS_STATE_STEP = 0 GSS_STATE_STEP = 0
GSS_STATE_WRAP = 1 GSS_STATE_WRAP = 1
def __init__(self, config, reposname, def __init__(self, repos):
username = None, password = None, hostname = None,
port = None, ssl = 1, maxconnections = 1, tunnel = None,
reference = '""', sslclientcert = None, sslclientkey = None,
sslcacertfile = None, idlefolders = []):
self.ui = getglobalui() self.ui = getglobalui()
self.reposname = reposname self.repos = repos
self.config = config self.config = repos.getconfig()
self.username = username self.tunnel = repos.getpreauthtunnel()
self.password = password self.usessl = repos.getssl()
self.username = repos.getuser()
self.password = None
self.passworderror = None self.passworderror = None
self.goodpassword = None self.goodpassword = None
self.hostname = hostname self.hostname = repos.gethost()
self.tunnel = tunnel self.port = repos.getport()
self.port = port if self.port == None:
self.usessl = ssl self.port = 993 if self.usessl else 143
self.sslclientcert = sslclientcert self.sslclientcert = repos.getsslclientcert()
self.sslclientkey = sslclientkey self.sslclientkey = repos.getsslclientkey()
self.sslcacertfile = sslcacertfile self.sslcacertfile = repos.getsslcacertfile()
if self.sslcacertfile is None:
self.verifycert = None # disable cert verification
self.delim = None self.delim = None
self.root = None self.root = None
if port == None: self.maxconnections = repos.getmaxconnections()
if ssl:
self.port = 993
else:
self.port = 143
self.maxconnections = maxconnections
self.availableconnections = [] self.availableconnections = []
self.assignedconnections = [] self.assignedconnections = []
self.lastowner = {} self.lastowner = {}
self.semaphore = BoundedSemaphore(self.maxconnections) self.semaphore = BoundedSemaphore(self.maxconnections)
self.connectionlock = Lock() self.connectionlock = Lock()
self.reference = reference self.reference = repos.getreference()
self.idlefolders = idlefolders self.idlefolders = repos.getidlefolders()
self.gss_step = self.GSS_STATE_STEP self.gss_step = self.GSS_STATE_STEP
self.gss_vc = None self.gss_vc = None
self.gssapi = False self.gssapi = False
def getpassword(self): def getpassword(self):
if self.goodpassword != None: """Returns the server password or None"""
if self.goodpassword != None: # use cached good one first
return self.goodpassword return self.goodpassword
if self.password != None and self.passworderror == None: if self.password != None and self.passworderror == None:
return self.password return self.password # non-failed preconfigured one
self.password = self.ui.getpass(self.reposname, # get 1) configured password first 2) fall back to asking via UI
self.config, self.password = self.repos.getpassword() or \
self.ui.getpass(self.repos.getname(), self.config,
self.passworderror) self.passworderror)
self.passworderror = None self.passworderror = None
return self.password return self.password
def getdelim(self): def getdelim(self):
@ -107,12 +109,15 @@ class IMAPServer:
return self.root return self.root
def releaseconnection(self, connection): def releaseconnection(self, connection, drop_conn=False):
"""Releases a connection, returning it to the pool.""" """Releases a connection, returning it to the pool.
:param drop_conn: If True, the connection will be released and
not be reused. This can be used to indicate broken connections."""
self.connectionlock.acquire() self.connectionlock.acquire()
self.assignedconnections.remove(connection) self.assignedconnections.remove(connection)
# Don't reuse broken connections # Don't reuse broken connections
if connection.Terminate: if connection.Terminate or drop_conn:
connection.logout() connection.logout()
else: else:
self.availableconnections.append(connection) self.availableconnections.append(connection)
@ -204,17 +209,21 @@ class IMAPServer:
success = 1 success = 1
elif self.usessl: elif self.usessl:
self.ui.connecting(self.hostname, self.port) self.ui.connecting(self.hostname, self.port)
imapobj = imaplibutil.WrappedIMAP4_SSL(self.hostname, self.port, fingerprint = self.repos.get_ssl_fingerprint()
self.sslclientkey, self.sslclientcert, imapobj = imaplibutil.WrappedIMAP4_SSL(self.hostname,
self.port,
self.sslclientkey,
self.sslclientcert,
self.sslcacertfile,
self.verifycert,
timeout=socket.getdefaulttimeout(), timeout=socket.getdefaulttimeout(),
cacertfile = self.sslcacertfile) fingerprint=fingerprint
)
else: else:
self.ui.connecting(self.hostname, self.port) self.ui.connecting(self.hostname, self.port)
imapobj = imaplibutil.WrappedIMAP4(self.hostname, self.port, imapobj = imaplibutil.WrappedIMAP4(self.hostname, self.port,
timeout=socket.getdefaulttimeout()) timeout=socket.getdefaulttimeout())
imapobj.mustquote = imaplibutil.mustquote
if not self.tunnel: if not self.tunnel:
try: try:
# Try GSSAPI and continue if it fails # Try GSSAPI and continue if it fails
@ -260,7 +269,6 @@ class IMAPServer:
except imapobj.error, val: except imapobj.error, val:
self.passworderror = str(val) self.passworderror = str(val)
raise raise
#self.password = None
if self.delim == None: if self.delim == None:
listres = imapobj.list(self.reference, '""')[1] listres = imapobj.list(self.reference, '""')[1]
@ -292,8 +300,6 @@ class IMAPServer:
error...""" error..."""
self.semaphore.release() self.semaphore.release()
#Make sure that this can be retried the next time...
self.passworderror = None
if(self.connectionlock.locked()): if(self.connectionlock.locked()):
self.connectionlock.release() self.connectionlock.release()
@ -304,20 +310,20 @@ class IMAPServer:
reason = "Could not resolve name '%s' for repository "\ reason = "Could not resolve name '%s' for repository "\
"'%s'. Make sure you have configured the ser"\ "'%s'. Make sure you have configured the ser"\
"ver name correctly and that you are online."%\ "ver name correctly and that you are online."%\
(self.hostname, self.reposname) (self.hostname, self.repos)
raise OfflineImapError(reason, severity) raise OfflineImapError(reason, severity)
elif SSLError and isinstance(e, SSLError) and e.errno == 1: elif SSLError and isinstance(e, SSLError) and e.errno == 1:
# SSL unknown protocol error # SSL unknown protocol error
# happens e.g. when connecting via SSL to a non-SSL service # happens e.g. when connecting via SSL to a non-SSL service
if self.port != 443: if self.port != 993:
reason = "Could not connect via SSL to host '%s' and non-s"\ reason = "Could not connect via SSL to host '%s' and non-s"\
"tandard ssl port %d configured. Make sure you connect"\ "tandard ssl port %d configured. Make sure you connect"\
" to the correct port." % (self.hostname, self.port) " to the correct port." % (self.hostname, self.port)
else: else:
reason = "Unknown SSL protocol connecting to host '%s' for"\ reason = "Unknown SSL protocol connecting to host '%s' for"\
"repository '%s'. OpenSSL responded:\n%s"\ "repository '%s'. OpenSSL responded:\n%s"\
% (self.hostname, self.reposname, e) % (self.hostname, self.repos, e)
raise OfflineImapError(reason, severity) raise OfflineImapError(reason, severity)
elif isinstance(e, socket.error) and e.args[0] == errno.ECONNREFUSED: elif isinstance(e, socket.error) and e.args[0] == errno.ECONNREFUSED:
@ -333,7 +339,8 @@ class IMAPServer:
if str(e)[:24] == "can't open socket; error": if str(e)[:24] == "can't open socket; error":
raise OfflineImapError("Could not connect to remote server '%s' "\ raise OfflineImapError("Could not connect to remote server '%s' "\
"for repository '%s'. Remote does not answer." "for repository '%s'. Remote does not answer."
% (self.hostname, self.reposname), severity) % (self.hostname, self.repos),
OfflineImapError.ERROR.REPO)
else: else:
# re-raise all other errors # re-raise all other errors
raise raise
@ -408,11 +415,56 @@ class IMAPServer:
self.ui.debug('imap', 'keepalive: bottom of loop') self.ui.debug('imap', 'keepalive: bottom of loop')
def verifycert(self, cert, hostname):
'''Verify that cert (in socket.getpeercert() format) matches hostname.
CRLs are not handled.
Returns error message if any problems are found and None on success.
'''
errstr = "CA Cert verifying failed: "
if not cert:
return ('%s no certificate received' % errstr)
dnsname = hostname.lower()
certnames = []
# cert expired?
notafter = cert.get('notAfter')
if notafter:
if time.time() >= cert_time_to_seconds(notafter):
return '%s certificate expired %s' % (errstr, notafter)
# First read commonName
for s in cert.get('subject', []):
key, value = s[0]
if key == 'commonName':
certnames.append(value.lower())
if len(certnames) == 0:
return ('%s no commonName found in certificate' % errstr)
# Then read subjectAltName
for key, value in cert.get('subjectAltName', []):
if key == 'DNS':
certnames.append(value.lower())
# And finally try to match hostname with one of these names
for certname in certnames:
if (certname == dnsname or
'.' in dnsname and certname == '*.' + dnsname.split('.', 1)[1]):
return None
return ('%s no matching domain name found in certificate' % errstr)
class IdleThread(object): class IdleThread(object):
def __init__(self, parent, folder=None): def __init__(self, parent, folder=None):
"""If invoked without 'folder', perform a NOOP and wait for
self.stop() to be called. If invoked with folder, switch to IDLE
mode and synchronize once we have a new message"""
self.parent = parent self.parent = parent
self.folder = folder self.folder = folder
self.event = Event() self.stop_sig = Event()
self.ui = getglobalui()
if folder is None: if folder is None:
self.thread = Thread(target=self.noop) self.thread = Thread(target=self.noop)
else: else:
@ -423,7 +475,7 @@ class IdleThread(object):
self.thread.start() self.thread.start()
def stop(self): def stop(self):
self.event.set() self.stop_sig.set()
def join(self): def join(self):
self.thread.join() self.thread.join()
@ -431,7 +483,7 @@ class IdleThread(object):
def noop(self): def noop(self):
imapobj = self.parent.acquireconnection() imapobj = self.parent.acquireconnection()
imapobj.noop() imapobj.noop()
self.event.wait() self.stop_sig.wait()
self.parent.releaseconnection(imapobj) self.parent.releaseconnection(imapobj)
def dosync(self): def dosync(self):
@ -446,87 +498,56 @@ class IdleThread(object):
ui.unregisterthread(currentThread()) ui.unregisterthread(currentThread())
def idle(self): def idle(self):
while True: """Invoke IDLE mode until timeout or self.stop() is invoked"""
if self.event.isSet():
return
self.needsync = False
self.imapaborted = False
def callback(args): def callback(args):
result, cb_arg, exc_data = args """IDLE callback function invoked by imaplib2
if exc_data is None:
if not self.event.isSet():
self.needsync = True
self.event.set()
else:
# We got an "abort" signal.
self.imapaborted = True
self.stop()
This is invoked when a) The IMAP server tells us something
while in IDLE mode, b) we get an Exception (e.g. on dropped
connections, or c) the standard imaplib IDLE timeout of 29
minutes kicks in."""
result, cb_arg, exc_data = args
if exc_data is None and not self.stop_sig.isSet():
# No Exception, and we are not supposed to stop:
self.needsync = True
self.stop_sig.set() # continue to sync
while not self.stop_sig.isSet():
self.needsync = False
success = False # successfully selected FOLDER?
while not success:
imapobj = self.parent.acquireconnection() imapobj = self.parent.acquireconnection()
try:
imapobj.select(self.folder) imapobj.select(self.folder)
except OfflineImapError, e:
if e.severity == OfflineImapError.ERROR.FOLDER_RETRY:
# Connection closed, release connection and retry
self.ui.error(e, exc_info()[2])
self.parent.releaseconnection(imapobj, True)
else:
raise e
else:
success = True
if "IDLE" in imapobj.capabilities: if "IDLE" in imapobj.capabilities:
imapobj.idle(callback=callback) imapobj.idle(callback=callback)
else: else:
ui = getglobalui() self.ui.warn("IMAP IDLE not supported on server '%s'."
ui.warn("IMAP IDLE not supported on connection to %s." "Sleep until next refresh cycle." % imapobj.identifier)
"Falling back to old behavior: sleeping until next"
"refresh cycle."
%(imapobj.identifier,))
imapobj.noop() imapobj.noop()
self.event.wait() self.stop_sig.wait() # self.stop() or IDLE callback are invoked
if self.event.isSet(): try:
# Can't NOOP on a bad connection. # End IDLE mode with noop, imapobj can point to a dropped conn.
if not self.imapaborted:
imapobj.noop() imapobj.noop()
# We don't do event.clear() so that we'll fall out except imapobj.abort():
# of the loop next time around. self.ui.warn('Attempting NOOP on dropped connection %s' % \
self.parent.releaseconnection(imapobj) imapobj.identifier)
if self.needsync: self.parent.releaseconnection(imapobj, True)
self.event.clear()
self.dosync()
class ConfigedIMAPServer(IMAPServer):
"""This class is designed for easier initialization given a ConfigParser
object and an account name. The passwordhash is used if
passwords for certain accounts are known. If the password for this
account is listed, it will be obtained from there."""
def __init__(self, repository, passwordhash = {}):
"""Initialize the object. If the account is not a tunnel,
the password is required."""
self.repos = repository
self.config = self.repos.getconfig()
usetunnel = self.repos.getpreauthtunnel()
if not usetunnel:
host = self.repos.gethost()
user = self.repos.getuser()
port = self.repos.getport()
ssl = self.repos.getssl()
sslclientcert = self.repos.getsslclientcert()
sslclientkey = self.repos.getsslclientkey()
sslcacertfile = self.repos.getsslcacertfile()
reference = self.repos.getreference()
idlefolders = self.repos.getidlefolders()
server = None
password = None
if repository.getname() in passwordhash:
password = passwordhash[repository.getname()]
# Connect to the remote server.
if usetunnel:
IMAPServer.__init__(self, self.config, self.repos.getname(),
tunnel = usetunnel,
reference = reference,
idlefolders = idlefolders,
maxconnections = self.repos.getmaxconnections())
else: else:
if not password: self.parent.releaseconnection(imapobj)
password = self.repos.getpassword()
IMAPServer.__init__(self, self.config, self.repos.getname(), if self.needsync:
user, password, host, port, ssl, # here not via self.stop, but because IDLE responded. Do
self.repos.getmaxconnections(), # another round and invoke actual syncing.
reference = reference, self.stop_sig.clear()
idlefolders = idlefolders, self.dosync()
sslclientcert = sslclientcert,
sslclientkey = sslclientkey,
sslcacertfile = sslcacertfile)

View File

@ -20,6 +20,11 @@ import re
import string import string
import types import types
from offlineimap.ui import getglobalui from offlineimap.ui import getglobalui
try: # python 2.6 has set() built in
set
except NameError:
from sets import Set as set
quotere = re.compile('^("(?:[^"]|\\\\")*")') quotere = re.compile('^("(?:[^"]|\\\\")*")')
def debug(*args): def debug(*args):
@ -42,11 +47,21 @@ def dequote(string):
return string return string
def flagsplit(string): def flagsplit(string):
"""Converts a string of IMAP flags to a list
:returns: E.g. '(\\Draft \\Deleted)' returns ['\\Draft','\\Deleted'].
(FLAGS (\\Seen Old) UID 4807) returns
['FLAGS,'(\\Seen Old)','UID', '4807']
"""
if string[0] != '(' or string[-1] != ')': if string[0] != '(' or string[-1] != ')':
raise ValueError, "Passed string '%s' is not a flag list" % string raise ValueError, "Passed string '%s' is not a flag list" % string
return imapsplit(string[1:-1]) return imapsplit(string[1:-1])
def options2hash(list): def options2hash(list):
"""convert list [1,2,3,4,5,6] to {1:2, 3:4, 5:6}"""
# effectively this does dict(zip(l[::2],l[1::2])), however
# measurements seemed to have indicated that the manual variant is
# faster for mosly small lists.
retval = {} retval = {}
counter = 0 counter = 0
while (counter < len(list)): while (counter < len(list)):
@ -55,8 +70,12 @@ def options2hash(list):
debug("options2hash returning:", retval) debug("options2hash returning:", retval)
return retval return retval
def flags2hash(string): def flags2hash(flags):
return options2hash(flagsplit(string)) """Converts IMAP response string from eg IMAP4.fetch() to a hash.
E.g. '(FLAGS (\\Seen Old) UID 4807)' leads to
{'FLAGS': '(\\Seen Old)', 'UID': '4807'}"""
return options2hash(flagsplit(flags))
def imapsplit(imapstring): def imapsplit(imapstring):
"""Takes a string from an IMAP conversation and returns a list containing """Takes a string from an IMAP conversation and returns a list containing
@ -152,15 +171,16 @@ flagmap = [('\\Seen', 'S'),
('\\Draft', 'D')] ('\\Draft', 'D')]
def flagsimap2maildir(flagstring): def flagsimap2maildir(flagstring):
retval = [] """Convert string '(\\Draft \\Deleted)' into a flags set(DR)"""
imapflaglist = [x.lower() for x in flagstring[1:-1].split()] retval = set()
imapflaglist = flagstring[1:-1].split()
for imapflag, maildirflag in flagmap: for imapflag, maildirflag in flagmap:
if imapflag.lower() in imapflaglist: if imapflag in imapflaglist:
retval.append(maildirflag) retval.add(maildirflag)
retval.sort()
return retval return retval
def flagsmaildir2imap(maildirflaglist): def flagsmaildir2imap(maildirflaglist):
"""Convert set of flags ([DR]) into a string '(\\Draft \\Deleted)'"""
retval = [] retval = []
for imapflag, maildirflag in flagmap: for imapflag, maildirflag in flagmap:
if maildirflag in maildirflaglist: if maildirflag in maildirflaglist:
@ -168,38 +188,32 @@ def flagsmaildir2imap(maildirflaglist):
retval.sort() retval.sort()
return '(' + ' '.join(retval) + ')' return '(' + ' '.join(retval) + ')'
def listjoin(list): def uid_sequence(uidlist):
start = None """Collapse UID lists into shorter sequence sets
end = None
retval = []
def getlist(start, end): [1,2,3,4,5,10,12,13] will return "1:5,10,12:13". This function sorts
the list, and only collapses if subsequent entries form a range.
:returns: The collapsed UID list as string"""
def getrange(start, end):
if start == end: if start == end:
return(str(start)) return(str(start))
else: return "%s:%s" % (start, end)
return(str(start) + ":" + str(end))
if not len(uidlist): return '' # Empty list, return
start, end = None, None
retval = []
# Force items to be longs and sort them
sorted_uids = sorted(map(int, uidlist))
for item in list: for item in iter(sorted_uids):
if start == None: item = int(item)
# First item. if start == None: # First item
start = item start, end = item, item
end = item elif item == end + 1: # Next item in a range
elif item == end + 1:
# An addition to the list.
end = item
else:
# Here on: starting a new list.
retval.append(getlist(start, end))
start = item
end = item end = item
else: # Starting a new range
retval.append(getrange(start, end))
start, end = item, item
if start != None: retval.append(getrange(start, end)) # Add final range/item
retval.append(getlist(start, end))
return ",".join(retval) return ",".join(retval)

View File

@ -114,37 +114,32 @@ class BaseRepository(object, CustomConfig.ConfigHelperMixin):
def getfolder(self, foldername): def getfolder(self, foldername):
raise NotImplementedError raise NotImplementedError
def syncfoldersto(self, dest, copyfolders): def syncfoldersto(self, dst_repo, status_repo):
"""Syncs the folders in this repository to those in dest. """Syncs the folders in this repository to those in dest.
It does NOT sync the contents of those folders.
For every time dest.makefolder() is called, also call makefolder() It does NOT sync the contents of those folders."""
on each folder in copyfolders.""" src_repo = self
src = self src_folders = src_repo.getfolders()
srcfolders = src.getfolders() dst_folders = dst_repo.getfolders()
destfolders = dest.getfolders()
# Create hashes with the names, but convert the source folders # Create hashes with the names, but convert the source folders
# to the dest folder's sep. # to the dest folder's sep.
src_hash = {}
srchash = {} for folder in src_folders:
for folder in srcfolders: src_hash[folder.getvisiblename().replace(
srchash[folder.getvisiblename().replace(src.getsep(), dest.getsep())] = \ src_repo.getsep(), dst_repo.getsep())] = folder
folder dst_hash = {}
desthash = {} for folder in dst_folders:
for folder in destfolders: dst_hash[folder.getvisiblename()] = folder
desthash[folder.getvisiblename()] = folder
# #
# Find new folders. # Find new folders.
# for key in src_hash.keys():
if not key in dst_hash:
for key in srchash.keys():
if not key in desthash:
try: try:
dest.makefolder(key) dst_repo.makefolder(key)
for copyfolder in copyfolders: status_repo.makefolder(key.replace(dst_repo.getsep(),
copyfolder.makefolder(key.replace(dest.getsep(), copyfolder.getsep())) status_repo.getsep()))
except (KeyboardInterrupt): except (KeyboardInterrupt):
raise raise
except: except:

View File

@ -24,6 +24,7 @@ from threading import Event
import re import re
import types import types
import os import os
from sys import exc_info
import netrc import netrc
import errno import errno
@ -33,7 +34,7 @@ class IMAPRepository(BaseRepository):
BaseRepository.__init__(self, reposname, account) BaseRepository.__init__(self, reposname, account)
# self.ui is being set by the BaseRepository # self.ui is being set by the BaseRepository
self._host = None self._host = None
self.imapserver = imapserver.ConfigedIMAPServer(self) self.imapserver = imapserver.IMAPServer(self)
self.folders = None self.folders = None
self.nametrans = lambda foldername: foldername self.nametrans = lambda foldername: foldername
self.folderfilter = lambda foldername: 1 self.folderfilter = lambda foldername: 1
@ -181,6 +182,9 @@ class IMAPRepository(BaseRepository):
% (self.name, cacertfile)) % (self.name, cacertfile))
return cacertfile return cacertfile
def get_ssl_fingerprint(self):
return self.getconf('cert_fingerprint', None)
def getpreauthtunnel(self): def getpreauthtunnel(self):
return self.getconf('preauthtunnel', None) return self.getconf('preauthtunnel', None)
@ -307,7 +311,12 @@ class IMAPRepository(BaseRepository):
for foldername in self.folderincludes: for foldername in self.folderincludes:
try: try:
imapobj.select(foldername, readonly = 1) imapobj.select(foldername, readonly = 1)
except ValueError: except OfflineImapError, e:
# couldn't select this folderinclude, so ignore folder.
if e.severity > OfflineImapError.ERROR.FOLDER:
raise
self.ui.error(e, exc_info()[2],
'Invalid folderinclude:')
continue continue
retval.append(self.getfoldertype()(self.imapserver, retval.append(self.getfoldertype()(self.imapserver,
foldername, foldername,

View File

@ -50,10 +50,17 @@ class LocalStatusRepository(BaseRepository):
return '.' return '.'
def getfolderfilename(self, foldername): def getfolderfilename(self, foldername):
"""Return the full path of the status file""" """Return the full path of the status file
# replace with 'dot' if final path name is '.'
foldername = re.sub('(^|\/)\.$','\\1dot', foldername) This mimics the path that Folder().getfolderbasename() would return"""
return os.path.join(self.directory, foldername) if not foldername:
basename = '.'
else: #avoid directory hierarchies and file names such as '/'
basename = foldername.replace('/', '.')
# replace with literal 'dot' if final path name is '.' as '.' is
# an invalid file name.
basename = re.sub('(^|\/)\.$','\\1dot', basename)
return os.path.join(self.directory, basename)
def makefolder(self, foldername): def makefolder(self, foldername):
"""Create a LocalStatus Folder """Create a LocalStatus Folder
@ -80,14 +87,13 @@ class LocalStatusRepository(BaseRepository):
self.config) self.config)
def getfolders(self): def getfolders(self):
"""Returns a list of ALL folders on this server. """Returns a list of all cached folders."""
This is currently nowhere used in the code."""
if self._folders != None: if self._folders != None:
return self._folders return self._folders
self._folders = []
for folder in os.listdir(self.directory): for folder in os.listdir(self.directory):
self._folders = retval.append(self.getfolder(folder)) self._folders.append(self.getfolder(folder))
return self._folders return self._folders
def forgetfolders(self): def forgetfolders(self):

View File

@ -54,9 +54,9 @@ class BlinkenBase:
s.gettf().setcolor('blue') s.gettf().setcolor('blue')
s.__class__.__bases__[-1].syncingmessages(s, sr, sf, dr, df) s.__class__.__bases__[-1].syncingmessages(s, sr, sf, dr, df)
def copyingmessage(s, uid, src, destlist): def copyingmessage(s, uid, src, destfolder):
s.gettf().setcolor('orange') s.gettf().setcolor('orange')
s.__class__.__bases__[-1].copyingmessage(s, uid, src, destlist) s.__class__.__bases__[-1].copyingmessage(s, uid, src, destfolder)
def deletingmessages(s, uidlist, destlist): def deletingmessages(s, uidlist, destlist):
s.gettf().setcolor('red') s.gettf().setcolor('red')

View File

@ -230,7 +230,7 @@ class CursesThreadFrame:
if self.getcolor() == 'black': if self.getcolor() == 'black':
self.window.addstr(self.y, self.x, ' ', self.color) self.window.addstr(self.y, self.x, ' ', self.color)
else: else:
self.window.addstr(self.y, self.x, self.ui.config.getdefault("ui.Curses.Blinkenlights", "statuschar", '.'), self.color) self.window.addstr(self.y, self.x, '.', self.color)
self.c.stdscr.move(self.c.height - 1, self.c.width - 1) self.c.stdscr.move(self.c.height - 1, self.c.width - 1)
self.window.refresh() self.window.refresh()
self.c.locked(lockedstuff) self.c.locked(lockedstuff)

View File

@ -108,10 +108,10 @@ class MachineUI(UIBase):
(s.getnicename(sr), sf.getname(), s.getnicename(dr), (s.getnicename(sr), sf.getname(), s.getnicename(dr),
df.getname())) df.getname()))
def copyingmessage(s, uid, src, destlist): def copyingmessage(self, uid, srcfolder, destfolder):
ds = s.folderlist(destlist) self._printData('copyingmessage', "%d\n%s\n%s\n%s[%s]" % \
s._printData('copyingmessage', "%d\n%s\n%s\n%s" % \ (uid, self.getnicename(srcfolder), srcfolder.getname(),
(uid, s.getnicename(src), src.getname(), ds)) self.getnicename(destfolder), destfolder))
def folderlist(s, list): def folderlist(s, list):
return ("\f".join(["%s\t%s" % (s.getnicename(x), x.getname()) for x in list])) return ("\f".join(["%s\t%s" % (s.getnicename(x), x.getname()) for x in list]))

View File

@ -21,6 +21,7 @@ import time
import sys import sys
import traceback import traceback
import threading import threading
from Queue import Queue
import offlineimap import offlineimap
debugtypes = {'':'Other offlineimap related sync messages', debugtypes = {'':'Other offlineimap related sync messages',
@ -47,6 +48,8 @@ class UIBase:
s.debugmsglen = 50 s.debugmsglen = 50
s.threadaccounts = {} s.threadaccounts = {}
s.logfile = None s.logfile = None
s.exc_queue = Queue()
"""saves all occuring exceptions, so we can output them at the end"""
################################################## UTILS ################################################## UTILS
def _msg(s, msg): def _msg(s, msg):
@ -82,6 +85,39 @@ class UIBase:
else: else:
s._msg("WARNING: " + msg) s._msg("WARNING: " + msg)
def error(self, exc, exc_traceback=None, msg=None):
"""Log a message at severity level ERROR
Log Exception 'exc' to error log, possibly prepended by a preceding
error "msg", detailing at what point the error occurred.
In debug mode, we also output the full traceback that occurred
if one has been passed in via sys.info()[2].
Also save the Exception to a stack that can be output at the end
of the sync run when offlineiamp exits. It is recommended to
always pass in exceptions if possible, so we can give the user
the best debugging info.
One example of such a call might be:
ui.error(exc, sys.exc_info()[2], msg="While syncing Folder %s in "
"repo %s")
"""
cur_thread = threading.currentThread()
if msg:
self._msg("ERROR [%s]: %s\n %s" % (cur_thread, msg, exc))
else:
self._msg("ERROR [%s]: %s" % (cur_thread, exc))
if not self.debuglist:
# only output tracebacks in debug mode
exc_traceback = None
# push exc on the queue for later output
self.exc_queue.put((msg, exc, exc_traceback))
if exc_traceback:
self._msg(traceback.format_tb(exc_traceback))
def registerthread(s, account): def registerthread(s, account):
"""Provides a hint to UIs about which account this particular """Provides a hint to UIs about which account this particular
thread is processing.""" thread is processing."""
@ -249,11 +285,12 @@ class UIBase:
s.getnicename(dr), s.getnicename(dr),
df.getname())) df.getname()))
def copyingmessage(s, uid, src, destlist): def copyingmessage(self, uid, src, destfolder):
if s.verbose >= 0: """Output a log line stating which message we copy"""
ds = s.folderlist(destlist) if self.verbose >= 0:
s._msg("Copy message %d %s[%s] -> %s" % (uid, s.getnicename(src), self._msg("Copy message %d %s[%s] -> %s[%s]" % \
src.getname(), ds)) (uid, self.getnicename(src), src,
self.getnicename(destfolder), destfolder))
def deletingmessage(s, uid, destlist): def deletingmessage(s, uid, destlist):
if s.verbose >= 0: if s.verbose >= 0:
@ -265,7 +302,7 @@ class UIBase:
ds = s.folderlist(destlist) ds = s.folderlist(destlist)
s._msg("Deleting %d messages (%s) in %s" % \ s._msg("Deleting %d messages (%s) in %s" % \
(len(uidlist), (len(uidlist),
", ".join([str(u) for u in uidlist]), offlineimap.imaputil.uid_sequence(uidlist),
ds)) ds))
def addingflags(s, uidlist, flags, dest): def addingflags(s, uidlist, flags, dest):
@ -315,12 +352,24 @@ class UIBase:
def mainException(s): def mainException(s):
s._msg(s.getMainExceptionString()) s._msg(s.getMainExceptionString())
def terminate(s, exitstatus = 0, errortitle = None, errormsg = None): def terminate(self, exitstatus = 0, errortitle = None, errormsg = None):
"""Called to terminate the application.""" """Called to terminate the application."""
if errormsg <> None: #print any exceptions that have occurred over the run
if errortitle <> None: if not self.exc_queue.empty():
sys.stderr.write('ERROR: %s\n\n%s\n'%(errortitle, errormsg)) self._msg("\nERROR: Exceptions occurred during the run!")
while not self.exc_queue.empty():
msg, exc, exc_traceback = self.exc_queue.get()
if msg:
self._msg("ERROR: %s\n %s" % (msg, exc))
else: else:
self._msg("ERROR: %s" % (exc))
if exc_traceback:
self._msg("\nTraceback:\n%s" %"".join(
traceback.format_tb(exc_traceback)))
if errormsg and errortitle:
sys.stderr.write('ERROR: %s\n\n%s\n'%(errortitle, errormsg))
elif errormsg:
sys.stderr.write('%s\n' % errormsg) sys.stderr.write('%s\n' % errormsg)
sys.exit(exitstatus) sys.exit(exitstatus)