Commit Graph

1847 Commits

Author SHA1 Message Date
Rodolfo García Peñas (kix)
33f3b4b1aa Do not use UNIX signals in Windows
This patch does not use the signals in Windows. just avoid these calls.

This patch is related to issue #37, issue 3.
2021-07-25 15:53:09 +02:00
Rodolfo García Peñas (kix)
09fc0ece61 imaplib2 version function
The new imaplib2 version >= 3.06 includes a function "version()",
and this patch uses the old style __version__ value and the new
function.

This patch is related to issue #37, issue 2.
2021-07-25 15:53:09 +02:00
Rodolfo García Peñas (kix)
a6fd6b1ded Include portalocker in imaplibutil.py
This patch include the usage of portalocker for Windows environments,
because fcntl is not available.

This patch is related to issue #37, issue 1.
2021-07-25 15:50:47 +02:00
Rodolfo García Peñas (kix)
fe443e6bd3
Merge pull request #72 from jishac/encoding_edge_cases
Encoding: edge cases and error handling
2021-06-09 06:03:54 +02:00
Joseph Ishac
9e0fb59bdf Adding missing import of NoBoundaryInMultipartDefect 2021-06-07 22:18:31 -04:00
Joseph Ishac
5b976fc5da Making error handling when parsing messages consistent 2021-06-07 21:57:54 -04:00
Joseph Ishac
9bebcbe4f7 Adding utilities to Base class.
Moving the quoted boundary fix to the Base class so that it can be used
by any subclass that needs to read an email.  Adding another utility to
extract message-id from a raw email.
2021-06-07 20:26:26 -04:00
Joseph Ishac
b4f100c92c Merge branch to add extra debugging to imap parsing into encoding_edge_cases branch 2021-06-07 14:07:43 -04:00
Joseph Ishac
b0aad911ab Adding some extra error handling when parsing from imaplib 2021-06-07 14:05:07 -04:00
Rodolfo García Peñas (kix)
721d579a32
Merge pull request #66 from duritong/fix-fingerprint-verification
disable verification of ssl certs via CA if fingerpint is set
2021-05-09 11:28:17 +02:00
Rodolfo García Peñas (kix)
f6848015ff
Merge pull request #55 from sudipm-mukherjee/gmail
BUG: Gmail FETCH error with synclabels enabled
2021-05-09 11:24:03 +02:00
Rodolfo García Peñas (kix)
edef557bf3
Merge pull request #54 from sudipm-mukherjee/curses
BUG: Right format for password from Curses
2021-05-09 11:23:24 +02:00
mh
871b1175f1 disable verification of ssl certs via CA if fingerpint is set
Up to a4863b2 offlineimap did not include a default CA bundle. And
folks who set cert_fingerprint (because they might connect to a
host using a self-signed cert or an onion service without the onion
address in the SANs) were able to validate their certificates.

Since a4863b2 you always have a `sslcacertfile` configured (since
it always falls back to the os one) and thus the old way didn't
work anymore.

If a use defines a `cert_fingerprint` there is not much use to
validate the cert through the CA chain, since the fingerprint
is the stronges verification you can get. Therefor we can disable
verfication when `cert_fingerprint` is set.

This enables users to fetch emails again from onion services or
hosts using self-signed certifcates, but doesn't question nor
change any other behavior.

Fixes #41
2021-04-29 14:34:06 +02:00
Joseph Ishac
84b96e1daa Adding re.DOTALL to catch potentially folded lines and fixed formatting 2021-04-14 17:08:16 -04:00
Joseph Ishac
a4532294ae Correcting an issue where dbg_output was not defined when the server was
unreachable due to an optimization in PR#56.  Since message-id is more
useful to better pin point the correct message, removing dbg_output.

Also fixing https://github.com/OfflineIMAP/offlineimap3/issues/62 by
correcting broken multipart boundaries or raising an error if as_bytes()
fails.  Related python bug submitted: https://bugs.python.org/issue43818
although this workaround should be sufficent in the interim.

Signed-off-by: Joseph Ishac <jishac@nasa.gov>
2021-04-14 14:54:25 -04:00
Joseph Ishac
f024bb9e4c Wrap bytes check in a wrapper to be a bit more efficient 2021-04-14 11:19:22 -04:00
Joseph Ishac
0345390aa1 Fixed oversight in regular expression (boundary cannot be empty) 2021-04-14 11:14:44 -04:00
Joseph Ishac
b78af75064 Adding a handler to detect the unlikely edge case where a message may
have an improperly quoted boundary that can cause the python library to
fail to reproduce the original message with msg.as_bytes().  See:
https://bugs.python.org/issue43818 and
https://github.com/OfflineIMAP/offlineimap3/issues/62
2021-04-13 00:01:26 -04:00
Joseph Ishac
6a45eef3b5 Fixed a minor bug discovered if server is unreachable when debugging 2021-04-12 22:58:58 -04:00
Joseph Ishac
7e7349d8ed Final Testing of IMAP and MailDir complete. GMAIL should be patched as well and I reviewed the code several times. However, I cannot test it, testers wanted!
This commit: Minor bug fixes from testing

Should finalize implementation of enhancement #48
https://github.com/OfflineIMAP/offlineimap3/issues/48

And fix issues #43 and #44
https://github.com/OfflineIMAP/offlineimap3/issues/43
https://github.com/OfflineIMAP/offlineimap3/issues/44

Signed-off-by: Joseph Ishac <jishac@nasa.gov>
Tested-by: Joseph Ishac <jishac@nasa.gov>
2021-02-23 20:12:55 -05:00
Joseph Ishac
259bf83607 Fixing up the rest of the parsing and IMAP functions, and GMAIL classes as well. Also adding is_debugging() to the UI to allow us to quickly determine if we should build some of the expensive debug objects 2021-02-23 16:17:54 -05:00
Joseph Ishac
5fc08e529b BUG: behavior of fetch now only returns a single entry, the use of
data[0][0] here is a carry over from the old behavior of offlineimap
(python2)
2021-02-22 23:27:45 -05:00
Joseph Ishac
3166723baa Removing obsolete emailutil.py. The date functionality was pulled into folder/Base.py which now handles messages as binary objects 2021-02-21 23:20:39 -05:00
Joseph Ishac
14b318b0e2 Formatting and adding the requirements that were removed as part of the "Included charset detection" patch that was reverted. Added all but chardet~=3.0.4 which is not needed with this fix. 2021-02-21 23:09:33 -05:00
Joseph Ishac
f71ef226bf Merge branch 'master' into multiple_encoding_support. Dropping patch
for "Included charset detection"
2021-02-21 22:36:30 -05:00
Joseph Ishac
c8b275cdc3 Revert "Included charset detection"
This reverts commit 62490ff183.
2021-02-21 22:31:49 -05:00
Sudip Mukherjee
8b88441759 BUG: Gmail FETCH error with synclabels enabled
If synclabels is enabled then offlineimap is sending '1:*' to imaplib2,
and imaplib2 while creating the FETCH command is quoting the sequence
and the command becomes:
b"JFFJ10 FETCH '1:*' (FLAGS X-GM-LABELS UID)\r\n"

Remove the single-quotes to prevent that and also consider the response
as bytes.

Closes: #52

Signed-off-by: Sudip Mukherjee <sudipm.mukherjee@gmail.com>
2021-02-20 00:34:01 +00:00
Sudip Mukherjee
3afd3395bd BUG: Exception with debug logs
When ui is set to 'Curses Blinkenlights' and debug logs are enabled,
we get an exception with 'embedded null character'.

Remove the NULL from the log, keeping the log message same as before.

Signed-off-by: Sudip Mukherjee <sudipm.mukherjee@gmail.com>
2021-02-20 00:20:36 +00:00
Sudip Mukherjee
c11232cf53 BUG: Right format for password from Curses
Reading the password from Curses Blinkenlights returns a bytes objects
instead an utf-8 string.

This patch will decode if bytes is provided.

Closes: #49

Signed-off-by: Sudip Mukherjee <sudipm.mukherjee@gmail.com>
2021-02-20 00:20:36 +00:00
Joseph Ishac
49b6c30ace Cleaning up some more use of the old message string "content" to use "msg" email object instead. 2021-02-19 17:00:15 -05:00
Rodolfo García Peñas (kix)
62490ff183 Included charset detection
This patch includes charset detection to read the message.

This patch is related to issue #43

Signed-off-by: Rodolfo García Peñas (kix) <kix@kix.es>
2021-02-19 16:39:17 +01:00
Rodolfo García Peñas (kix)
76c7a723db
Merge pull request #51 from sudipm-mukherjee/fix_remotepassfile
Right format for password from remotepassfile
2021-02-18 18:40:06 +01:00
Rodolfo García Peñas (kix)
9e6d614cee
Merge pull request #45 from agx/fix-gssapi
imapserver: GSSAPI: make sure reply is all bytes
2021-02-18 18:39:22 +01:00
Rodolfo García Peñas (kix)
18c6612587
Merge pull request #50 from knatsakis/knats-fix-remoteusereval
BUG: Right format for username using remoteusereval
2021-02-18 18:37:57 +01:00
Sudip Mukherjee
96793820de BUG: Right format for password from remotepassfile
Reading the password using remotepassfile returns a bytes objects
instead an utf-8 string.

This patch includes support strings and bytes objects.

Closes: #40

Signed-off-by: Sudip Mukherjee <sudipm.mukherjee@gmail.com>
2021-02-16 19:41:08 +00:00
Joseph Ishac
1d2478bcb6 Series of *UNTESTED* changes that should move the internal structure of
a message from a string to an email object that is part of the built-in
email library.  The allows for emails to be processed as bytes and
re-encoded properly if they are not UTF-8 or ascii encoded.  Currently
these changes cover the Base, IMAP, and Maildir classes but not the
specialized GMAIL class yet.
2021-02-09 14:58:30 -05:00
Konstantinos Natsakis
fa080b8d92
BUG: Right format for username using remoteusereval
Similarly to 7a4285370f, reading the username
using remoteusereval returns a bytes objects instead an utf-8 string.

This patch includes support for both string and bytes objects.

Signed-off-by: Konstantinos Natsakis <5933427+knatsakis@users.noreply.github.com>
2021-02-08 23:54:50 +02:00
Amit Ramon
910e194a99 Fix utf8foldernames configuration option to work with Python3
The current code for supporting the utf8foldernames does not work with
Python3 due to changes in the 'codecs' module and related functions.

This fix adapts the existing code to work with Python3.

Signed-off-by: Amit Ramon <amit@riseup.net>
2021-02-05 16:57:51 +02:00
Guido Günther
d19024ef20 imapserver: GSSAPI: make sure reply is all bytes
The current code mixed string and bytes leading to:

ERROR: Exceptions occurred during the run!
ERROR: While attempting to sync account 'honk.sigxcpu.org'
  sequence item 1: expected str instance, int found

Traceback:
  File "/usr/share/offlineimap3/offlineimap/accounts.py", line 298, in syncrunner
    self.__sync()
  File "/usr/share/offlineimap3/offlineimap/accounts.py", line 374, in __sync
    remoterepos.getfolders()
  File "/usr/share/offlineimap3/offlineimap/repository/IMAP.py", line 648, in getfolders
    imapobj = self.imapserver.acquireconnection()
  File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 592, in acquireconnection
    self.__authn_helper(imapobj)
  File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 449, in __authn_helper
    if func(imapobj):
  File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 362, in __authn_gssapi
    imapobj.authenticate('GSSAPI', self.__gsshandler)
  File "/usr/lib/python3/dist-packages/imaplib2.py", line 691, in authenticate
    typ, dat = self._simple_command('AUTHENTICATE', mechanism.upper())
  File "/usr/lib/python3/dist-packages/imaplib2.py", line 1684, in _simple_command
    return self._command_complete(self._command(name, *args), kw)
  File "/usr/lib/python3/dist-packages/imaplib2.py", line 1404, in _command
    literal = literator(data, rqb)
  File "/usr/lib/python3/dist-packages/imaplib2.py", line 2247, in process
    ret = self.mech(self.decode(data))
  File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 318, in __gsshandler
    reply = ''.join(reply)

Closes: #46
Signed-off-by: Guido Günther <agx@sigxcpu.org>
2021-02-03 20:34:48 +01:00
Thomas De Schampheleire
58d34df29f imapserver: fix exception handling in xoauth2handler
In case of an exception in XOAUTH2 code refresh (HTTP Error 400: Bad
Request), following exception occurs:

ERROR: While attempting to sync account 'xxx'
  __init__() missing 3 required positional arguments: 'msg', 'hdrs', and 'fp'

Traceback:
  File ".../offlineimap3/offlineimap/accounts.py", line 298, in syncrunner
    self.__sync()
  File ".../offlineimap3/offlineimap/accounts.py", line 374, in __sync
    remoterepos.getfolders()
  File ".../offlineimap3/offlineimap/repository/IMAP.py", line 648, in getfolders
    imapobj = self.imapserver.acquireconnection()
  File ".../offlineimap3/offlineimap/imapserver.py", line 591, in acquireconnection
    self.__authn_helper(imapobj)
  File ".../offlineimap3/offlineimap/imapserver.py", line 448, in __authn_helper
    if func(imapobj):
  File ".../offlineimap3/offlineimap/imapserver.py", line 382, in __authn_xoauth2
    imapobj.authenticate('XOAUTH2', self.__xoauth2handler)
  File ".../offlineimap3/venv/lib/python3.7/site-packages/imaplib2.py", line 682, in authenticate
    typ, dat = self._simple_command('AUTHENTICATE', mechanism.upper())
  File ".../offlineimap3/venv/lib/python3.7/site-packages/imaplib2.py", line 1675, in _simple_command
    return self._command_complete(self._command(name, *args), kw)
  File ".../offlineimap3/venv/lib/python3.7/site-packages/imaplib2.py", line 1395, in _command
    literal = literator(data, rqb)
  File ".../offlineimap3/venv/lib/python3.7/site-packages/imaplib2.py", line 2238, in process
    ret = self.mech(self.decode(data))
  File ".../offlineimap3/offlineimap/imapserver.py", line 257, in __xoauth2handler
    raise type(e)(msg, exc_info()[2])

The exception 'e' is of type HTTPError, which does not have the same kind of
constructor as normal Python exceptions.

Instead, print the constructed message and just raise the existing
exception.

With that change, the same condition triggers another problem further on:

ERROR: While attempting to sync account 'xxx'
  tuple index out of range

Traceback:
  File ".../offlineimap3/offlineimap/accounts.py", line 298, in syncrunner
    self.__sync()
  File ".../offlineimap3/offlineimap/accounts.py", line 374, in __sync
    remoterepos.getfolders()
  File ".../offlineimap3/offlineimap/repository/IMAP.py", line 648, in getfolders
    imapobj = self.imapserver.acquireconnection()
  File ".../offlineimap3/offlineimap/imapserver.py", line 664, in acquireconnection
    elif isinstance(e, socket.error) and e.args[0] == errno.ECONNREFUSED:

because e.args is empty.
2021-01-04 10:26:01 +01:00
Rodolfo García Peñas (kix)
63ce3f770d foldersort broken with instance comparison
This patch solves a problem about the comparison of mails used in
foldersort.

When foldersort is used, for example with:

 lambda x,y: -cmp(x,y)

The user gets an error:

ERROR: While attempting to sync account 'accountname'
  '<' not supported between instances of 'K' and 'K'

Traceback:
  File "offlineimap3/offlineimap/accounts.py", line 298, in syncrunner
    self.__sync()
  File "offlineimap3/offlineimap/accounts.py", line 374, in __sync
    remoterepos.getfolders()
  File "offlineimap3/offlineimap/repository/IMAP.py", line 725, in getfolders
    retval.sort(key=cmp2key(self.foldersort))

The problem is because in Python 3, we must specify all parameters for
total_ordering (see https://docs.python.org/3/library/functools.html in
the functools.total_ordering block):

  Given a class defining one or more rich comparison ordering methods,
  this class decorator supplies the rest. This simplifies the effort
  involved in specifying all of the possible rich comparison operations:

  The class must define one of __lt__(), __le__(), __gt__(), or __ge__().
  In addition, the class should supply an __eq__() method.

Also, see: https://docs.python.org/3.1/library/stdtypes.html#comparisons

  Instances of a class cannot be ordered with respect to other instances
  of the same class, or other types of object, unless the class defines
  enough of the methods __lt__(), __le__(), __gt__(), and __ge__()
  (in general, __lt__() and __eq__() are sufficient, if you want the
  conventional meanings of the comparison operators).

This patch implements all methods.

Closes #33
2020-12-30 21:49:19 +01:00
Rodolfo García Peñas (kix)
63a1d865a8 BUG: offlineimap3 fails with tunneled connections
Running offlineimap with tunneled connections fails.

The connection to the server is right, the server sends the
PREAUTH message, offlineimap reads it and provides a reply, something
like 'UID1 CAPABILITY'. This message is added to the output queue in
imaplib2.py, function _command():

 if literal is None:
     self.ouq.put(rqb)
     return rqb

Then, the function _writer() in imaplib2 calls the self.send() function:

 try:
     self.send(rqb.data)
     if __debug__: self._log(4, '> %r' % rqb.data)

self object is an IMAP4_Tunnel class, and the function send() writes the
message, but the message is not sent to the server.

We need flush the buffer.

Closes #30
2020-12-28 18:54:53 +01:00
Rodolfo García Peñas (kix)
0fe1caa6a5 Avoid crash in search if no results
I am incluiding this check to avoid crash if the array is empty
and we are trying to read the first element.
2020-11-08 22:07:14 +01:00
Rodolfo García Peñas (kix)
ca0a2651a3 Added Debug for imaplib2
When I ported offlineimap from Python 2 to Python 3 I removed the Debug for
IMAP (imaplib2).
The reason was offlineimap was setting the Debug directly in imaplib2,
not using the proper way (using the IMAP4 argument). Because we are
removing the virtual_imaplib2, I removed this option.

I removed this line in offlineimap/init.py:303:

---8<---
 300        dtype = dtype.strip()
 301        self.ui.add_debug(dtype)
 302        if dtype.lower() == u'imap':
-303          imaplib.Debug = 5
 304
 305     if options.runonce:
 306         # Must kill the possible default option.
---8<---

With this patch, the debug level 5 is restored in imaplib if the user
set the -d ALL or -d imap in offlineimap.
2020-11-08 21:30:22 +01:00
Rodolfo García Peñas (kix)
a311faf183 IMAP search now works fine
This patch converts the search results from bytes to strings

I add a bit comment about it here:

    In Py2, with IMAP, imaplib2 returned a list of one element string.
      ['1, 2, 3, ...'] -> in Py3 is [b'1 2 3,...']
    In Py2, with Davmail, imaplib2 returned a list of strings.
      ['1', '2', '3', ...] -> in Py3 should be [b'1', b'2', b'3',...]

    In my tests with Py3, I get a list with one element: [b'1 2 3 ...']
    Then I convert the values to string and I get ['1 2 3 ...']

    With Davmail, it should be [b'1', b'2', b'3',...]
    When I convert the values to string, I get ['1', '2', '3',...]
2020-11-08 15:47:51 +01:00
Rodolfo García Peñas (kix)
5ccb89a412 BUG: Read response as string from APPENDUID
We need read the response from APPENUID and convert it to string.
This patch do it.
2020-11-07 16:48:09 +01:00
Rodolfo García Peñas (kix)
c130e84a84 BUG: Support for server capabilities
Server capabilities are returned as list of bytes. We need convert them
to list of strings.

This patch do it. Probably you must recreate your cache after this patch.
2020-11-07 15:52:06 +01:00
Rodolfo García Peñas (kix)
3f86218e55 IMAP.py split long lines
This patch split long lines (>=80 chars)
2020-11-07 15:25:27 +01:00
Rodolfo García Peñas (kix)
7a4285370f BUG: Right format for password using remotepasseval
Reading the password using remotepasseval returns a bytes objects
instead an utf-8 string.

This patch includes support strings and bytes objects.

Closes #21

    Signed-off-by: Rodolfo García Peñas (kix) <kix@kix.es>
2020-11-04 22:17:32 +01:00
Rodolfo García Peñas (kix)
f7534c70ce __init__.py added docstrings
Just added docstrings.
2020-11-01 13:12:03 +01:00