Commit Graph

2494 Commits

Author SHA1 Message Date
Sebastian Spaeth
47390e03d6 Don't CHECK imapserver after each APPEND
Most servers support the UIDPLUS extension, and we don't have to search
headers after each uploaded message. There is no need to CHECK the imap
server after each message when there is no need to search headers.

I have not measured the performance impact on real world servers, but
this lets us do less unneeded work in the common case.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-20 14:43:12 +01:00
Sebastian Spaeth
4d47f7bf3c Clean up and improve APPENDUID handling
We were not cleaning out possibly existing APPENDUID messages before
APPENDing a new message. In case an old message were still hanging
around, this could *possibly* lead to retrieving and old UID. Things
should have been fine, but we do want to play safe here.

Also, make use of the "official" imaplib2 .response() command rather
than the internal _get_untagged_response() function.

Remove the hack that we would be looking for APPENDUID responses even if
the server claimed not to support the UIDPLUS ext. We now poll server
CAPABILITIES after login, and Gmail does provide us with the UIDPLUS
capability after login.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-20 14:33:16 +01:00
Sebastian Spaeth
36dd46abfc Fix warn() output for MachineUI
We used "self" but the paramter was called "s". Fixes a crash when we
ui.warn() (only when using the MachineUI).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-20 11:33:30 +01:00
Sebastian Spaeth
09ef5ab042 Remove stray debug line
This line was accidentally left laying around.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-20 11:31:41 +01:00
Sebastian Spaeth
5fe522ae4f Merge branch 'master' into next
Conflicts:
	offlineimap/accounts.py
	offlineimap/folder/Base.py
	offlineimap/ui/Machine.py
	offlineimap/ui/UIBase.py
2012-01-20 11:28:59 +01:00
Sebastian Spaeth
fae48f309e Release 6.2.1-rc1
See Changelog.rst for details

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-20 11:10:42 +01:00
Sebastian Spaeth
eb04036da5 Make folders containing quotes work
imaputil.imapsplit did not cope with strings that contained encoded
quotation marks, e.g. a folder name '"Make" Magazine' would fail and
crash OfflineImap. Make it work by adapting the regex that we use to
extract the first quote to also work with encoded \" quotes. (We do no
sanity checks that there is an even number of such marks within a string
though)

This commit makes such folders work. This was reported and analyzed by
Mark Eichin.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>

Conflicts:

	Changelog.draft.rst
2012-01-20 11:07:33 +01:00
Sebastian Spaeth
c7df907cf6 Make folders containing quotes work
imaputil.imapsplit did not cope with strings that contained encoded
quotation marks, e.g. a folder name '"Make" Magazine' would fail and
crash OfflineImap. Make it work by adapting the regex that we use to
extract the first quote to also work with encoded \" quotes. (We do no
sanity checks that there is an even number of such marks within a string
though)

This commit makes such folders work. This was reported and analyzed by
Mark Eichin.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-20 11:06:20 +01:00
Daniel Shahaf
bf31accb78 We have a (read-only) commit mailing list
Sebastian Spaeth wrote on Thu, Jan 19, 2012 at 10:28:19 +0100:
> Forgot to Cc the list...
>
> On Wed, 18 Jan 2012 08:17:44 +0200, Daniel Shahaf wrote:
> > Sebastian Spaeth wrote on Wed, Jan 18, 2012 at 01:02:22 +0100:
> >> http://docs.offlineimap.org
> >
> > I've subscribed.  And if you tell me where the sources for docs.o.o are,
> > I'll send a patch for them, too. :-)
>
> They are autogenerated from docs/dev-docs-src, and
> docs/[INSTALL|MANUAL|FAQ].rst respectively. Patches are welcome...
>
> Sebastian

From 84fcb9fa5de9eb2f95d588a7403d9c6d13f0c7f0 Mon Sep 17 00:00:00 2001
From: Daniel Shahaf <d.s@daniel.shahaf.name>
Date: Thu, 19 Jan 2012 11:53:13 +0200
Subject: [PATCH] Document the commits@ list

Mention the commits@ list in SubmittingPatches.rst.

Mention SubmittingPatches.rst in the API part of the documentation.

Signed-off-by: Daniel Shahaf <d.s@daniel.shahaf.name>
2012-01-20 00:42:58 +01:00
Sebastian Spaeth
d2f8504757 Fix getuidvalidity crash (UIDVALIDITY returning None)
Rename getuidvalidity -> get_uidvalidity and cache the IMAP result.

1) Start modernizing our function names using more underscores
2) IMAPs implementation of get_uidvalidity was removing the UIDVALIDITY result
   from the imaplib2 result stack, so subsequent calls would return "None".
   As various functions can invoke this, this led to some errors that we
   avoid by caching the current UIDVALIDITY value in the Folder instance.

There are more simplifications and improvements to be made.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:09:37 +01:00
Sebastian Spaeth
165cf5cb4f Use modern "with open() as file"
Saves a LOC and guarantees that the file will be closed

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:03:03 +01:00
Sebastian Spaeth
de1e03b065 Tweak code documentation
Minor comment changes

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:03:03 +01:00
Sebastian Spaeth
d3c693be7d Rename folder.saveuidvalidity()
Simply rename to folder.save_uidvalidity()

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:03:03 +01:00
Sebastian Spaeth
eb616da602 Rename folder.isuidvalidityok()
Use nicer name folder.check_uidvalidity() and improve code documentation

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:03:03 +01:00
Sebastian Spaeth
297a618441 Rename folder.getsaveduidvalidity(), folder.getuidvalidity()
Modernize names to: folder.get_saveduidvalidity(), folder.get_uidvalidity()

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:03:03 +01:00
Sebastian Spaeth
9453e1d955 Fix getuidvalidity crash (UIDVALIDITY returning None)
Rename getuidvalidity -> get_uidvalidity and cache the IMAP result.

1) Start modernizing our function names using more underscores
2) IMAPs implementation of get_uidvalidity was removing the UIDVALIDITY result
   from the imaplib2 result stack, so subsequent calls would return "None".
   As various functions can invoke this, this led to some errors that we
   avoid by caching the current UIDVALIDITY value in the Folder instance.

There are more simplifications and improvements to be made.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:02:13 +01:00
Sebastian Spaeth
7c4fea906f Changelog entry for the test suite
Announce existence of a test suite, although it is still quite useless.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 10:34:19 +01:00
Sebastian Spaeth
2776ee1974 Create temp directories in "test" dir
Don't create the temp dirs in the current directory, but create them in
the same directory as the credentials.conf file is. We still need to
prevent deleting them on test failure, so that they can be inspected
manually.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 10:32:05 +01:00
Sebastian Spaeth
bf1f79c571 Add "test" command to setup.py
"python setup.py test" will now run the complete test suite. Remove the
previous ./test command.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 10:32:05 +01:00
Sebastian Spaeth
1f91d0fd06 Adapt imports so that test/OLItest can be imported
To run the test suite from the main setup.py we need to be able to
import and run the test suite from a different folder than the "test"
dir.

Make "test" a package and fix the imports to still work when imported
from another folder.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 10:32:05 +01:00
Sebastian Spaeth
1cdabcec46 Add code comment
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 10:32:05 +01:00
Sebastian Spaeth
9a905188f2 Add userfriendly warning if credentials.conf is missing
Warn the user and advise what she should be doing to get going.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 10:32:05 +01:00
Sebastian Spaeth
7b75e04b5b Add initial skeleton for a testing framework
This is the first revision that actually performs a test, in that it
starts up OfflineImap and sees if there are any exceptions.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 10:32:05 +01:00
Sebastian Spaeth
c1625aa719 DOCS: Tweak SSL FAQ a bit more
Cleanup the previously added information.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 10:15:04 +01:00
Sebastian Spaeth
2c19a63b9d DOCS: Add SSL debugging info to FAQ
Take the debugging instructions by Daniel Shahaf and add them to the
FAQ.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 10:08:31 +01:00
Sebastian Spaeth
075c1d2cdc TTYUI: Fix missing "time" import
commit f2a94af5 introduced the use of time.sleep in ui/TTY.py without
importing it. This caused a regression in 6.5.2, crashing OfflineIMap
when in refresh mode.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 02:04:25 +01:00
Sebastian Spaeth
3cd3edefca TTYUI: Fix python 2.6 compatibility
We were using super() on a class derived from logging.Formatter() which
worked fine in python 2.7. Apparently python 2.6 uses old-style classes
for this, so the TTYUI broke and crashed OfflineImap. This was
introduced in OLI 6.5.0, I think.

Fix it by calling logging.Formatter.... directly, rather than the
elegant super() (which I happen to like a lot more than is appropriate
in the python world).

Reported by Nik Reiman as github issue 23, should fix that issue.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 02:04:16 +01:00
Sebastian Spaeth
48802ae4b6 TTYUI: Fix missing "time" import
commit f2a94af5 introduced the use of time.sleep in ui/TTY.py without
importing it. This caused a regression in 6.5.2, crashing OfflineIMap
when in refresh mode.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 02:02:55 +01:00
Sebastian Spaeth
344b2f0b7a TTYUI: Fix python 2.6 compatibility
We were using super() on a class derived from logging.Formatter() which
worked fine in python 2.7. Apparently python 2.6 uses old-style classes
for this, so the TTYUI broke and crashed OfflineImap. This was
introduced in OLI 6.5.0, I think.

Fix it by calling logging.Formatter.... directly, rather than the
elegant super() (which I happen to like a lot more than is appropriate
in the python world).

Reported by Nik Reiman as github issue 23, should fix that issue.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-18 00:39:04 +01:00
Sebastian Spaeth
73b2b01dfa Changelog entry for previous commit
Forgot to add a changelog entry.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-17 05:14:50 +01:00
Sebastian Spaeth
6dc74c9da5 Improve delete performance with SQLITE backend
When deleting many (eg 2000) mails using the SQLITE backend, this takes
a long time durig which OfflineImap can not be aborted via
CTRL-C. Thinking it had frozen permanently, I killed it hard, leaving a
corrupted db journal (which leads to awkwards complaints by OLI on
subsequent starts!). That shows that delete performance is critical and
needs improvement.

We were iterating through the list of messages to delete and deleted
them one-by-one execute()'ing a new SQL Query for each message. This
patch improves the situation by allowing us to use executemany(), which
is -despite still being one SQL query per message- much faster. This is
because rather than performing a commit() after each mail, we now do
only one commit() after all mails have been deleted.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-17 04:17:15 +01:00
Sebastian Spaeth
f7866a880a Merge branch 'next' 2012-01-17 01:40:41 +01:00
Sebastian Spaeth
7a5768e471 Release v6.5.2
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-17 01:40:12 +01:00
Sebastian Spaeth
340c2f9a89 Improve documentation
Improve wording, and integrate the MANUAL and INSTALL and FAQ documents
into our complete user manual that is hosted online.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-17 01:34:56 +01:00
Sebastian Spaeth
f2a94af522 Don't emit regular sleeping announcements in Basic UI
They are too noisy in the Basic UI, but are still included in TTYUI.
Reported by John Wiegley.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-10 07:20:49 +01:00
Sebastian Spaeth
1b30ab896e Add changelog entry
Changelog entry for commit 0b103e8627

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-09 16:39:50 +01:00
Cédric Krier
3028648e8f Don't fail if /etc/netrc is not readable 2012-01-09 16:39:12 +01:00
Sebastian Spaeth
62b8027dfe Merge pull request #21 from cedk/master
Fix issue20: don't fail if netrc is not readable by us
2012-01-09 07:36:32 -08:00
Sebastian Spaeth
08c21248eb Merge branch 'next' 2012-01-09 10:04:54 +01:00
Sebastian Spaeth
9e65cfca21 Release v6.5.2-rc1
OfflineIMAP v6.5.2-rc1 (2012-01-09)
===================================
Commits v6.5.1.1 - v6.5.2-rc1:
note: Proper Changelog still in Changelog-draft.rst
d72bb88 Improve error message
3284e01 Revert "use .response() rather _get_untagged_response()"
81f194a mbnames should write out local and not nametransformed box names
7184ec2 Sanity check return value of UIDVALIDTY response
50de217 Allow to pass 'force' arg to selectro() to enforce a new select
ed71805 Changelog entry about "realdelete" option
0a275b9 Add scary warnings about "realdelete" option

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-09 10:04:05 +01:00
Sebastian Spaeth
d72bb88729 Improve error message
Add *what* UID was returned in case savemessage did not return a UID>0

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-09 09:57:36 +01:00
Sebastian Spaeth
3284e010ff Revert "use .response() rather _get_untagged_response()"
Recently the internal function use of imaplib2's _get_untagged_response()
was switched to use the public documented .response() function (which
should return the same data). However within a few fays we received reports
that both uses of a) the UIDVALIDITY fetching and b) the APPENDUID fetching
returned [None] as data although the IMAP log definitely shows that data
was returned. Revert to using the undocumented internal imaplib2 function,
that seemed to have worked without problems. This needs to be taken up to
the imaplib2 developer.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-09 09:51:43 +01:00
Cédric Krier
0b103e8627 Don't fail if /etc/netrc is not readable 2012-01-08 19:57:03 +01:00
Sebastian Spaeth
81f194adca mbnames should write out local and not nametransformed box names
Rather than to write out the nametrans'lated folder names for mbnames,
we now write out the local untransformed box names. This is generally
what we want. This became relevant since we support nametrans rules on
the local side since only a short time. Reported by Paul Collignan.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-08 12:48:21 +01:00
Sebastian Spaeth
7184ec28cc Sanity check return value of UIDVALIDTY response
We have a reported case where response('UIDVALIDITY') returned [None]
which results in an ugly non-intuitive crash. Sanity check and report
something nicer.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-08 12:26:47 +01:00
Sebastian Spaeth
50de2174bf Allow to pass 'force' arg to selectro() to enforce a new select
Pass through the 'force' argument from selectro() to select() so that it
can also enforce a new SELECT even if we already are on that folder.

Also change the default parameter from '0' to 'False' to make clear that
this is a Bool.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-08 11:29:54 +01:00
Sebastian Spaeth
ed71805476 Changelog entry about "realdelete" option
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-08 01:13:08 +01:00
Sebastian Spaeth
b5a13157e4 Add scary warnings about "realdelete" option
WARNING: I consider the Gmail "realdelete" option as harmful with the
potential for DATALOSS. Add scary warnings to offlineimap.conf.

See the analysis at
http://article.gmane.org/gmane.mail.imap.offlineimap.general/5265

Deleting a message from a Gmail folder via the IMAP interface will just
remove that folder's label from the message: the message will continue
to exist in the '[Gmail]/All Mail' folder.  If `realdelete` is set to
`True`, then deleted messages will be moved to the '[Gmail]/Trash'
folder.  BEWARE: this will immediately delete a messages from *all
folders* it belongs to!

AS OFFLINEIMAP IMPLEMENTS FOLDER MOVES AS 1) AN ADD and 2) A DELETE (the
order can vary), THIS MEANS THAT A FOLDER MOVE CAN CAUSE DATALOSS. DO
NOT USE IT AND MOVE MAIL TO "[Gmail]/Trash" TO DELETE MAIL FROM
"[Gmail]/All Mail"!

We will need to discuss whether to completely disable that feature.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-08 01:11:24 +01:00
Sebastian Spaeth
0a275b9532 Add scary warnings about "realdelete" option
WARNING: I consider the Gmail "realdelete" option as harmful with the
potential for DATALOSS. Add scary warnings to offlineimap.conf.

See the analysis at
http://article.gmane.org/gmane.mail.imap.offlineimap.general/5265

Deleting a message from a Gmail folder via the IMAP interface will just
remove that folder's label from the message: the message will continue
to exist in the '[Gmail]/All Mail' folder.  If `realdelete` is set to
`True`, then deleted messages will be moved to the '[Gmail]/Trash'
folder.  BEWARE: this will immediately delete a messages from *all
folders* it belongs to!

AS OFFLINEIMAP IMPLEMENTS FOLDER MOVES AS 1) AN ADD and 2) A DELETE (the
order can vary), THIS MEANS THAT A FOLDER MOVE CAN CAUSE DATALOSS. DO
NOT USE IT AND MOVE MAIL TO "[Gmail]/Trash" TO DELETE MAIL FROM
"[Gmail]/All Mail"!

We will need to discuss whether to completely disable that feature.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-08 01:06:56 +01:00
Sebastian Spaeth
15b4a7a5b9 OfflineIMAP v6.5.1.2
See changelog for details

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-07 23:34:00 +01:00