Reading the password using remotepassfile returns a bytes objects
instead an utf-8 string.
This patch includes support strings and bytes objects.
Closes: #40
Signed-off-by: Sudip Mukherjee <sudipm.mukherjee@gmail.com>
a message from a string to an email object that is part of the built-in
email library. The allows for emails to be processed as bytes and
re-encoded properly if they are not UTF-8 or ascii encoded. Currently
these changes cover the Base, IMAP, and Maildir classes but not the
specialized GMAIL class yet.
Similarly to 7a4285370f, reading the username
using remoteusereval returns a bytes objects instead an utf-8 string.
This patch includes support for both string and bytes objects.
Signed-off-by: Konstantinos Natsakis <5933427+knatsakis@users.noreply.github.com>
The current code for supporting the utf8foldernames does not work with
Python3 due to changes in the 'codecs' module and related functions.
This fix adapts the existing code to work with Python3.
Signed-off-by: Amit Ramon <amit@riseup.net>
The current code mixed string and bytes leading to:
ERROR: Exceptions occurred during the run!
ERROR: While attempting to sync account 'honk.sigxcpu.org'
sequence item 1: expected str instance, int found
Traceback:
File "/usr/share/offlineimap3/offlineimap/accounts.py", line 298, in syncrunner
self.__sync()
File "/usr/share/offlineimap3/offlineimap/accounts.py", line 374, in __sync
remoterepos.getfolders()
File "/usr/share/offlineimap3/offlineimap/repository/IMAP.py", line 648, in getfolders
imapobj = self.imapserver.acquireconnection()
File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 592, in acquireconnection
self.__authn_helper(imapobj)
File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 449, in __authn_helper
if func(imapobj):
File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 362, in __authn_gssapi
imapobj.authenticate('GSSAPI', self.__gsshandler)
File "/usr/lib/python3/dist-packages/imaplib2.py", line 691, in authenticate
typ, dat = self._simple_command('AUTHENTICATE', mechanism.upper())
File "/usr/lib/python3/dist-packages/imaplib2.py", line 1684, in _simple_command
return self._command_complete(self._command(name, *args), kw)
File "/usr/lib/python3/dist-packages/imaplib2.py", line 1404, in _command
literal = literator(data, rqb)
File "/usr/lib/python3/dist-packages/imaplib2.py", line 2247, in process
ret = self.mech(self.decode(data))
File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 318, in __gsshandler
reply = ''.join(reply)
Closes: #46
Signed-off-by: Guido Günther <agx@sigxcpu.org>
In case of an exception in XOAUTH2 code refresh (HTTP Error 400: Bad
Request), following exception occurs:
ERROR: While attempting to sync account 'xxx'
__init__() missing 3 required positional arguments: 'msg', 'hdrs', and 'fp'
Traceback:
File ".../offlineimap3/offlineimap/accounts.py", line 298, in syncrunner
self.__sync()
File ".../offlineimap3/offlineimap/accounts.py", line 374, in __sync
remoterepos.getfolders()
File ".../offlineimap3/offlineimap/repository/IMAP.py", line 648, in getfolders
imapobj = self.imapserver.acquireconnection()
File ".../offlineimap3/offlineimap/imapserver.py", line 591, in acquireconnection
self.__authn_helper(imapobj)
File ".../offlineimap3/offlineimap/imapserver.py", line 448, in __authn_helper
if func(imapobj):
File ".../offlineimap3/offlineimap/imapserver.py", line 382, in __authn_xoauth2
imapobj.authenticate('XOAUTH2', self.__xoauth2handler)
File ".../offlineimap3/venv/lib/python3.7/site-packages/imaplib2.py", line 682, in authenticate
typ, dat = self._simple_command('AUTHENTICATE', mechanism.upper())
File ".../offlineimap3/venv/lib/python3.7/site-packages/imaplib2.py", line 1675, in _simple_command
return self._command_complete(self._command(name, *args), kw)
File ".../offlineimap3/venv/lib/python3.7/site-packages/imaplib2.py", line 1395, in _command
literal = literator(data, rqb)
File ".../offlineimap3/venv/lib/python3.7/site-packages/imaplib2.py", line 2238, in process
ret = self.mech(self.decode(data))
File ".../offlineimap3/offlineimap/imapserver.py", line 257, in __xoauth2handler
raise type(e)(msg, exc_info()[2])
The exception 'e' is of type HTTPError, which does not have the same kind of
constructor as normal Python exceptions.
Instead, print the constructed message and just raise the existing
exception.
With that change, the same condition triggers another problem further on:
ERROR: While attempting to sync account 'xxx'
tuple index out of range
Traceback:
File ".../offlineimap3/offlineimap/accounts.py", line 298, in syncrunner
self.__sync()
File ".../offlineimap3/offlineimap/accounts.py", line 374, in __sync
remoterepos.getfolders()
File ".../offlineimap3/offlineimap/repository/IMAP.py", line 648, in getfolders
imapobj = self.imapserver.acquireconnection()
File ".../offlineimap3/offlineimap/imapserver.py", line 664, in acquireconnection
elif isinstance(e, socket.error) and e.args[0] == errno.ECONNREFUSED:
because e.args is empty.
This patch solves a problem about the comparison of mails used in
foldersort.
When foldersort is used, for example with:
lambda x,y: -cmp(x,y)
The user gets an error:
ERROR: While attempting to sync account 'accountname'
'<' not supported between instances of 'K' and 'K'
Traceback:
File "offlineimap3/offlineimap/accounts.py", line 298, in syncrunner
self.__sync()
File "offlineimap3/offlineimap/accounts.py", line 374, in __sync
remoterepos.getfolders()
File "offlineimap3/offlineimap/repository/IMAP.py", line 725, in getfolders
retval.sort(key=cmp2key(self.foldersort))
The problem is because in Python 3, we must specify all parameters for
total_ordering (see https://docs.python.org/3/library/functools.html in
the functools.total_ordering block):
Given a class defining one or more rich comparison ordering methods,
this class decorator supplies the rest. This simplifies the effort
involved in specifying all of the possible rich comparison operations:
The class must define one of __lt__(), __le__(), __gt__(), or __ge__().
In addition, the class should supply an __eq__() method.
Also, see: https://docs.python.org/3.1/library/stdtypes.html#comparisons
Instances of a class cannot be ordered with respect to other instances
of the same class, or other types of object, unless the class defines
enough of the methods __lt__(), __le__(), __gt__(), and __ge__()
(in general, __lt__() and __eq__() are sufficient, if you want the
conventional meanings of the comparison operators).
This patch implements all methods.
Closes#33
Running offlineimap with tunneled connections fails.
The connection to the server is right, the server sends the
PREAUTH message, offlineimap reads it and provides a reply, something
like 'UID1 CAPABILITY'. This message is added to the output queue in
imaplib2.py, function _command():
if literal is None:
self.ouq.put(rqb)
return rqb
Then, the function _writer() in imaplib2 calls the self.send() function:
try:
self.send(rqb.data)
if __debug__: self._log(4, '> %r' % rqb.data)
self object is an IMAP4_Tunnel class, and the function send() writes the
message, but the message is not sent to the server.
We need flush the buffer.
Closes#30
When I ported offlineimap from Python 2 to Python 3 I removed the Debug for
IMAP (imaplib2).
The reason was offlineimap was setting the Debug directly in imaplib2,
not using the proper way (using the IMAP4 argument). Because we are
removing the virtual_imaplib2, I removed this option.
I removed this line in offlineimap/init.py:303:
---8<---
300 dtype = dtype.strip()
301 self.ui.add_debug(dtype)
302 if dtype.lower() == u'imap':
-303 imaplib.Debug = 5
304
305 if options.runonce:
306 # Must kill the possible default option.
---8<---
With this patch, the debug level 5 is restored in imaplib if the user
set the -d ALL or -d imap in offlineimap.
This patch converts the search results from bytes to strings
I add a bit comment about it here:
In Py2, with IMAP, imaplib2 returned a list of one element string.
['1, 2, 3, ...'] -> in Py3 is [b'1 2 3,...']
In Py2, with Davmail, imaplib2 returned a list of strings.
['1', '2', '3', ...] -> in Py3 should be [b'1', b'2', b'3',...]
In my tests with Py3, I get a list with one element: [b'1 2 3 ...']
Then I convert the values to string and I get ['1 2 3 ...']
With Davmail, it should be [b'1', b'2', b'3',...]
When I convert the values to string, I get ['1', '2', '3',...]
Server capabilities are returned as list of bytes. We need convert them
to list of strings.
This patch do it. Probably you must recreate your cache after this patch.
Reading the password using remotepasseval returns a bytes objects
instead an utf-8 string.
This patch includes support strings and bytes objects.
Closes#21
Signed-off-by: Rodolfo García Peñas (kix) <kix@kix.es>
Remove the warnings:
repository/__init__.py:66:12: W0707: Consider explicitly re-raising using the 'from' keyword (raise-missing-from)
repository/__init__.py:74:12: W0707: Consider explicitly re-raising using the 'from' keyword (raise-missing-from)
Variable bk is renamed to bkend to avoid this pylint warning:
repository/LocalStatus.py:88:12: C0103: Variable name "bk" doesn't conform to snake_case naming style (invalid-name)
The function sync_folder_structure is only used in the accounts.py file:
kix@inle:~/src/offlineimap3/offlineimap$ rgrep sync_folder_structure *
accounts.py: remoterepos.sync_folder_structure(localrepos, statusrepos)
repository/Base.py: def sync_folder_structure(self, local_repo, status_repo):
kix@inle:~/src/offlineimap3/offlineimap$
The returned value is not used. We have this warning:
R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
Because is not returning anything here, but is returning None at the function end.
We can include this None here safely.
First, in Python 3 we don't need call the super().__init__ explicitly, because Python does it. So, we can remove the line:
super().__init__(reposname, account)
If we leafe the method __init__ empty, the parent __init__ method is not called, we need remove the __init__ and then Python uses the parent __init__ method.
The variable f is renamed to l_file to avoid pylint warning:
utils/distro_utils.py:95:8: C0103: Variable name "f" doesn't conform to snake_case naming style (invalid-name)
This patch renames some variables to avoid pylint warning about
Variable name "f" doesn't conform to snake_case naming style (invalid-name)
f is now file
th is now the_en
n is now count
Call to OfflineImapError has the arguments:
- message
- severity
- error code
The None argument ir wrong here, we can remove it and then we have three arguments not four.
This patch updates the cram-md5 auth. We include two steps:
- Convert the password variable from string to bytes. This change is
because in Python2 strings and bytes are the same, but not in Python3
- Updates the call to hmac.new, now the digestmod argument is mandatory.
I used hashlib.md5, because we need md5 hash.
Closes#19
Signed-off-by: Rodolfo García Peñas (kix) <kix@kix.es>
This patch includes a new functionality to get the sslcacertfile.
With this patch is possible to remove the sslcacertfile variable form the
.offlineimaprc and use the system's default. This could be easy for new users.
With this patch, the user can use ssl without the sslcacertfile option
in .offlineimaprc, use sslcacertfile = <empty> (no include anything
after the equal sign, os specify the sslcacertfile as previosly.
Currently, these are the options:
User specifies sslacertfile -> OK
User specifies sslacertfile=OS-DEFAULT
If the system's default is provided -> OK
If the system's default is not provided -> Fail
User do not specifies sslacertfile -> Fail
User specifies empty sslacertfile -> Fail
With the new option (this patch).
User specifies sslacertfile -> OK
User specifies sslacertfile=OS-DEFAULT or
User do not specifies sslacertfile or
User specifies empty sslacertfile
If the system's default is provided -> OK
If the system's default is not provided -> Fail
Closes#14
imaplib2 is doing this code for strings:
if isinstance(message, str):
message = bytes(message, 'ASCII')
But our message is already encoded using 'utf-8'.
Then, we can set the message as bytes, encoded using 'utf-8'
in offlineimap and imaplib2 won't change our message.
This patch solves this problem:
WARNING:OfflineImap:
Traceback:
File "/home/kix/src/offlineimap3/offlineimap/folder/Base.py", line 1127, in syncmessagesto
action(dstfolder, statusfolder)
File "/home/kix/src/offlineimap3/offlineimap/folder/Base.py", line 955, in __syncmessagesto_copy
self.copymessageto(uid, dstfolder, statusfolder, register=0)
File "/home/kix/src/offlineimap3/offlineimap/folder/Base.py", line 855, in copymessageto
new_uid = dstfolder.savemessage(uid, message, flags, rtime)
File "/home/kix/src/offlineimap3/offlineimap/folder/IMAP.py", line 668, in savemessage
(typ, dat) = imapobj.append(self.getfullIMAPname(),
File "/usr/lib/python3/dist-packages/imaplib2.py", line 660, in append
message = bytes(message, 'ASCII')
This patch changes the __DEF_OS_LOCATIONS to an iterable value.
The reason is because a list is an iterable, but an string is an
iterable too, and this is a mistake.
The function get_os_sslcertfile() has a loop to iterate the return of
get_os_sslcertfile_searchpath(), that returns the value in the
__DEF_OS_LOCATIONS dictionary. When the value is an iterable, the "f"
variable is set to the iterable value and works fine.
If the value of "f" is an string, the for-loop iterates over every
character, so the test for os.path.exists is always false (is comparing
the path with a character, not with the full path), so this function
fails and return None.
To check this change, edit your .offlineimaprc file and change the
sslcacertfile to OS-DEFAULT:
sslcacertfile = OS-DEFAULT
And run offlineimap. If you are not using 'darwin0 (the only iterable)
it will fails. Now, apply this patch, and run offlineimap again. Problem
is solved.
Signed-off-by: Rodolfo García Peñas (kix) <kix@kix.es>
The call from platform.linux_distribution was depdrecated in Python 3.7.
This patch solves this problem, using the recomended package distro.
Signed-off-by: Rodolfo García Peñas (kix) <kix@kix.es>
The file utils/distro.py is renamed to utils/distro_utils.py
because we need import the distro file to update some calls.
The file is used in IMAP.py, I need change the import.
Emails received may not be UTF-8. Following error was observed on a specific
mail:
Traceback (most recent call last):
File "/home/tdescham/repo/offlineimap3/offlineimap/threadutil.py", line 146, in run
Thread.run(self)
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/home/tdescham/repo/offlineimap3/offlineimap/folder/Base.py", line 850, in copymessageto
message = self.getmessage(uid)
File "/home/tdescham/repo/offlineimap3/offlineimap/folder/IMAP.py", line 327, in getmessage
data = self._fetch_from_imap(str(uid), self.retrycount)
File "/home/tdescham/repo/offlineimap3/offlineimap/folder/IMAP.py", line 844, in _fetch_from_imap
ndata1 = data[0][1].decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 10177: invalid start byte
This completely aborted offlineimap3, blocking further mail reception.
Instead, use the 'replace' error strategy in Python:
Replace with a suitable replacement character; Python will use the
official U+FFFD REPLACEMENT CHARACTER for the built-in Unicode codecs on
decoding and ‘?’ on encoding.
https://docs.python.org/2/library/codecs.html#codec-base-classes
ERROR: ERROR in syncfolder for gmail folder INBOX: Traceback (most recent call last):
File ".../offlineimap3/offlineimap/accounts.py", line 634, in syncfolder
cachemessagelists_upto_date(maxage)
File ".../offlineimap3/offlineimap/accounts.py", line 526, in cachemessagelists_upto_date
min_date=time.gmtime(time.mktime(date) + 24 * 60 * 60))
File ".../offlineimap3/offlineimap/folder/IMAP.py", line 277, in cachemessagelist
imapobj, min_date=min_date, min_uid=min_uid)
File ".../offlineimap3/offlineimap/folder/IMAP.py", line 259, in _msgs_to_fetch
search_result = search(search_cond)
File ".../offlineimap3/offlineimap/folder/IMAP.py", line 222, in search
if ' ' in res_data[0] or res_data[0] == '':
TypeError: a bytes-like object is required, not 'str'
When a hook is configured, the output is a bytes object.
This is then printed on the console/logfile as:
2020-10-12 08:36:17 INFO: Hook stdout: b'Processed 3 total files in almost no time.\nAdded 3 new messages to the database.\n'
Hook stderr:b''
Decode the output so that it is printed nicely, as:
2020-10-12 08:36:17 INFO: Hook stdout: Processed 3 total files in almost no time.
Added 3 new messages to the database.
Hook stderr:
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Following error is seen when parsing server responses for sent mail:
2020-10-12 08:19:11 WARNING: Can't parse FETCH response, we awaited string: b' UID 26855)'
2020-10-12 08:19:11 WARNING: savemessage: Searching mails for new Message-ID failed. Could not determine new UID on Sent.
The comparison with 'type("")' means comparing with 'string' type in Python
3, but the left-hand side is a bytes object.
In case a tuple was received (first case in the code), the input is already
decoded from bytes to strings, but in case a single input was received it
was not.
Note that the comparison with 'type("")' is a bit odd, a more logical way
seems to be:
if isinstance(item, bytes)
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
This patch allows using folders with atom-specials like
"(", ")", spaces,...
We need quotes the folder name if it includes this special
characters.
Closes#4
This patch remves the set() mutable argument as default value,
sets the default value to None and check if the argument is none
in the code to call set().
Error when using the XOAUTH2 token refresh logic:
("POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str. (configuration is: {....}", <traceback object at 0x7fc6e69f2b40>)
[' File ".../offlineimap3/offlineimap/accounts.py", line 298, in syncrunner
self.__sync()
', ' File ".../offlineimap3/offlineimap/accounts.py", line 374, in __sync
remoterepos.getfolders()
', ' File ".../offlineimap3/offlineimap/repository/IMAP.py", line 446, in getfolders
imapobj = self.imapserver.acquireconnection()
', ' File ".../offlineimap3/offlineimap/imapserver.py", line 579, in acquireconnection
self.__authn_helper(imapobj)
', ' File ".../offlineimap3/offlineimap/imapserver.py", line 443, in __authn_helper
if func(imapobj):
', ' File ".../offlineimap3/offlineimap/imapserver.py", line 377, in __authn_xoauth2
imapobj.authenticate(\'XOAUTH2\', self.__xoauth2handler)
', ' File ".../offlineimap3/venv/lib/python3.7/site-packages/imaplib2.py", line 681, in authenticate
typ, dat = self._simple_command(\'AUTHENTICATE\', mechanism.upper())
', ' File ".../offlineimap3/venv/lib/python3.7/site-packages/imaplib2.py", line 1674, in _simple_command
return self._command_complete(self._command(name, *args), kw)
', ' File ".../offlineimap3/venv/lib/python3.7/site-packages/imaplib2.py", line 1394, in _command
literal = literator(data, rqb)
', ' File ".../offlineimap3/venv/lib/python3.7/site-packages/imaplib2.py", line 2237, in process
ret = self.mech(self.decode(data))
', ' File ".../offlineimap3/offlineimap/imapserver.py", line 253, in __xoauth2handler
raise type(e)(msg, exc_info()[2])
']
Fix by encoding the data passed to urllib.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
This patch removes the library six, compatible with python2.
I need change these re-raise calls.
Signed-off-by: Rodolfo García Peñas (kix) <kix@kix.es>
