Commit Graph

531 Commits

Author SHA1 Message Date
Joseph Ishac
9bebcbe4f7 Adding utilities to Base class.
Moving the quoted boundary fix to the Base class so that it can be used
by any subclass that needs to read an email.  Adding another utility to
extract message-id from a raw email.
2021-06-07 20:26:26 -04:00
Joseph Ishac
b4f100c92c Merge branch to add extra debugging to imap parsing into encoding_edge_cases branch 2021-06-07 14:07:43 -04:00
Joseph Ishac
b0aad911ab Adding some extra error handling when parsing from imaplib 2021-06-07 14:05:07 -04:00
Joseph Ishac
84b96e1daa Adding re.DOTALL to catch potentially folded lines and fixed formatting 2021-04-14 17:08:16 -04:00
Joseph Ishac
a4532294ae Correcting an issue where dbg_output was not defined when the server was
unreachable due to an optimization in PR#56.  Since message-id is more
useful to better pin point the correct message, removing dbg_output.

Also fixing https://github.com/OfflineIMAP/offlineimap3/issues/62 by
correcting broken multipart boundaries or raising an error if as_bytes()
fails.  Related python bug submitted: https://bugs.python.org/issue43818
although this workaround should be sufficent in the interim.

Signed-off-by: Joseph Ishac <jishac@nasa.gov>
2021-04-14 14:54:25 -04:00
Joseph Ishac
f024bb9e4c Wrap bytes check in a wrapper to be a bit more efficient 2021-04-14 11:19:22 -04:00
Joseph Ishac
0345390aa1 Fixed oversight in regular expression (boundary cannot be empty) 2021-04-14 11:14:44 -04:00
Joseph Ishac
b78af75064 Adding a handler to detect the unlikely edge case where a message may
have an improperly quoted boundary that can cause the python library to
fail to reproduce the original message with msg.as_bytes().  See:
https://bugs.python.org/issue43818 and
https://github.com/OfflineIMAP/offlineimap3/issues/62
2021-04-13 00:01:26 -04:00
Joseph Ishac
6a45eef3b5 Fixed a minor bug discovered if server is unreachable when debugging 2021-04-12 22:58:58 -04:00
Joseph Ishac
7e7349d8ed Final Testing of IMAP and MailDir complete. GMAIL should be patched as well and I reviewed the code several times. However, I cannot test it, testers wanted!
This commit: Minor bug fixes from testing

Should finalize implementation of enhancement #48
https://github.com/OfflineIMAP/offlineimap3/issues/48

And fix issues #43 and #44
https://github.com/OfflineIMAP/offlineimap3/issues/43
https://github.com/OfflineIMAP/offlineimap3/issues/44

Signed-off-by: Joseph Ishac <jishac@nasa.gov>
Tested-by: Joseph Ishac <jishac@nasa.gov>
2021-02-23 20:12:55 -05:00
Joseph Ishac
259bf83607 Fixing up the rest of the parsing and IMAP functions, and GMAIL classes as well. Also adding is_debugging() to the UI to allow us to quickly determine if we should build some of the expensive debug objects 2021-02-23 16:17:54 -05:00
Joseph Ishac
5fc08e529b BUG: behavior of fetch now only returns a single entry, the use of
data[0][0] here is a carry over from the old behavior of offlineimap
(python2)
2021-02-22 23:27:45 -05:00
Joseph Ishac
3166723baa Removing obsolete emailutil.py. The date functionality was pulled into folder/Base.py which now handles messages as binary objects 2021-02-21 23:20:39 -05:00
Joseph Ishac
14b318b0e2 Formatting and adding the requirements that were removed as part of the "Included charset detection" patch that was reverted. Added all but chardet~=3.0.4 which is not needed with this fix. 2021-02-21 23:09:33 -05:00
Joseph Ishac
49b6c30ace Cleaning up some more use of the old message string "content" to use "msg" email object instead. 2021-02-19 17:00:15 -05:00
Joseph Ishac
1d2478bcb6 Series of *UNTESTED* changes that should move the internal structure of
a message from a string to an email object that is part of the built-in
email library.  The allows for emails to be processed as bytes and
re-encoded properly if they are not UTF-8 or ascii encoded.  Currently
these changes cover the Base, IMAP, and Maildir classes but not the
specialized GMAIL class yet.
2021-02-09 14:58:30 -05:00
Rodolfo García Peñas (kix)
0fe1caa6a5 Avoid crash in search if no results
I am incluiding this check to avoid crash if the array is empty
and we are trying to read the first element.
2020-11-08 22:07:14 +01:00
Rodolfo García Peñas (kix)
a311faf183 IMAP search now works fine
This patch converts the search results from bytes to strings

I add a bit comment about it here:

    In Py2, with IMAP, imaplib2 returned a list of one element string.
      ['1, 2, 3, ...'] -> in Py3 is [b'1 2 3,...']
    In Py2, with Davmail, imaplib2 returned a list of strings.
      ['1', '2', '3', ...] -> in Py3 should be [b'1', b'2', b'3',...]

    In my tests with Py3, I get a list with one element: [b'1 2 3 ...']
    Then I convert the values to string and I get ['1 2 3 ...']

    With Davmail, it should be [b'1', b'2', b'3',...]
    When I convert the values to string, I get ['1', '2', '3',...]
2020-11-08 15:47:51 +01:00
Rodolfo García Peñas (kix)
5ccb89a412 BUG: Read response as string from APPENDUID
We need read the response from APPENUID and convert it to string.
This patch do it.
2020-11-07 16:48:09 +01:00
Rodolfo García Peñas (kix)
3f86218e55 IMAP.py split long lines
This patch split long lines (>=80 chars)
2020-11-07 15:25:27 +01:00
Rodolfo García Peñas (kix)
f7534c70ce __init__.py added docstrings
Just added docstrings.
2020-11-01 13:12:03 +01:00
Rodolfo García Peñas (kix)
f53bee7335 Base.py added docstrings
Just added docstrings.
2020-11-01 13:11:52 +01:00
Rodolfo García Peñas (kix)
9239c804e9 Base.py remove redundant parenthesis
Just remove these parenthesis
2020-10-31 16:55:58 +01:00
Rodolfo García Peñas (kix)
8504cdc6b8 Base.py long lines style
This patch changes long lines (>80 chars) to lines <80 chars.
The patch only changes the style.
2020-10-31 16:55:20 +01:00
Rodolfo García Peñas (kix)
442c88d838 imaplib expect bytes in the append
imaplib2 is doing this code for strings:

        if isinstance(message, str):
            message = bytes(message, 'ASCII')

But our message is already encoded using 'utf-8'.
Then, we can set the message as bytes, encoded using 'utf-8'
in offlineimap and imaplib2 won't change our message.

This patch solves this problem:

WARNING:OfflineImap:
Traceback:
  File "/home/kix/src/offlineimap3/offlineimap/folder/Base.py", line 1127, in syncmessagesto
    action(dstfolder, statusfolder)
  File "/home/kix/src/offlineimap3/offlineimap/folder/Base.py", line 955, in __syncmessagesto_copy
    self.copymessageto(uid, dstfolder, statusfolder, register=0)
  File "/home/kix/src/offlineimap3/offlineimap/folder/Base.py", line 855, in copymessageto
    new_uid = dstfolder.savemessage(uid, message, flags, rtime)
  File "/home/kix/src/offlineimap3/offlineimap/folder/IMAP.py", line 668, in savemessage
    (typ, dat) = imapobj.append(self.getfullIMAPname(),
  File "/usr/lib/python3/dist-packages/imaplib2.py", line 660, in append
    message = bytes(message, 'ASCII')
2020-10-25 20:36:07 +01:00
Thomas De Schampheleire
33e0efa163 IMAP: replace non-UTF-8 characters rather than aborting
Emails received may not be UTF-8. Following error was observed on a specific
mail:

Traceback (most recent call last):
  File "/home/tdescham/repo/offlineimap3/offlineimap/threadutil.py", line 146, in run
    Thread.run(self)
  File "/usr/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/tdescham/repo/offlineimap3/offlineimap/folder/Base.py", line 850, in copymessageto
    message = self.getmessage(uid)
  File "/home/tdescham/repo/offlineimap3/offlineimap/folder/IMAP.py", line 327, in getmessage
    data = self._fetch_from_imap(str(uid), self.retrycount)
  File "/home/tdescham/repo/offlineimap3/offlineimap/folder/IMAP.py", line 844, in _fetch_from_imap
    ndata1 = data[0][1].decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 10177: invalid start byte

This completely aborted offlineimap3, blocking further mail reception.

Instead, use the 'replace' error strategy in Python:

    Replace with a suitable replacement character; Python will use the
    official U+FFFD REPLACEMENT CHARACTER for the built-in Unicode codecs on
    decoding and ‘?’ on encoding.
    https://docs.python.org/2/library/codecs.html#codec-base-classes
2020-10-21 16:29:13 +02:00
Thomas De Schampheleire
820e5c855f IMAP: Python 3 bytes fix on first download of account
ERROR: ERROR in syncfolder for gmail folder INBOX: Traceback (most recent call last):
  File ".../offlineimap3/offlineimap/accounts.py", line 634, in syncfolder
    cachemessagelists_upto_date(maxage)
  File ".../offlineimap3/offlineimap/accounts.py", line 526, in cachemessagelists_upto_date
    min_date=time.gmtime(time.mktime(date) + 24 * 60 * 60))
  File ".../offlineimap3/offlineimap/folder/IMAP.py", line 277, in cachemessagelist
    imapobj, min_date=min_date, min_uid=min_uid)
  File ".../offlineimap3/offlineimap/folder/IMAP.py", line 259, in _msgs_to_fetch
    search_result = search(search_cond)
  File ".../offlineimap3/offlineimap/folder/IMAP.py", line 222, in search
    if ' ' in res_data[0] or res_data[0] == '':
TypeError: a bytes-like object is required, not 'str'
2020-10-21 14:28:23 +02:00
Rodolfo García Peñas (kix)
49c85d732d Using isinstance instead type
This patch uses isinstance, like Thomas pointed in their last commit.
2020-10-12 12:52:04 +02:00
Rodolfo García Peñas (kix)
d8149c1ec1
Merge pull request #7 from patrickdepinguin/py3fix
More python 3 related fixes
2020-10-12 12:44:19 +02:00
Rodolfo García Peñas (kix)
ba6857bc87 Remove email.Parse
The Parse funcion was deprecated in python 2.4. We can use this new
code.
2020-10-12 12:40:54 +02:00
Rodolfo García Peñas (kix)
62df6e70c3 Rigth comparison with err EEXIST
This is the right code. The OSError doesn't have the EEXIST variable.
2020-10-12 12:35:56 +02:00
Thomas De Schampheleire
423785725b IMAP.py: server responses are in bytes, not string
Following error is seen when parsing server responses for sent mail:

2020-10-12 08:19:11 WARNING: Can't parse FETCH response, we awaited string: b' UID 26855)'
2020-10-12 08:19:11 WARNING: savemessage: Searching mails for new Message-ID failed. Could not determine new UID on Sent.

The comparison with 'type("")' means comparing with 'string' type in Python
3, but the left-hand side is a bytes object.

In case a tuple was received (first case in the code), the input is already
decoded from bytes to strings, but in case a single input was received it
was not.

Note that the comparison with 'type("")' is a bit odd, a more logical way
seems to be:
    if isinstance(item, bytes)

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
2020-10-12 09:01:51 +02:00
Rodolfo García Peñas (kix)
3d5f926552 Changed the doxygen in change_message_uid
This patch changes the doxygen and adds the ruid argument.
2020-10-11 23:57:17 +02:00
Rodolfo García Peñas (kix)
01c621d86c Allow folder names with atom specials
This patch allows using folders with atom-specials like
"(", ")", spaces,...

We need quotes the folder name if it includes this special
characters.

Closes #4
2020-10-11 23:01:08 +02:00
Rodolfo García Peñas (kix)
bb05904cf1 Renamed variable already used
The variable l was used in tn the code, so we should change it in
this code block.
2020-10-10 17:24:18 +02:00
Rodolfo García Peñas (kix)
659b38bcf7 Base.py Comment unused code in savemessagelabels
This code is not used, we can comment it.
2020-10-10 17:23:05 +02:00
Rodolfo García Peñas (kix)
7ff65be690 Removed mutable arguments
This patch remves the set() mutable argument as default value,
sets the default value to None and check if the argument is none
in the code to call set().
2020-10-10 15:00:34 +02:00
Rodolfo García Peñas (kix)
9123cb83fc Remove md5 import
The library md5 is only for python2, hashlib is used in python3

Signed-off-by: Rodolfo García Peñas (kix) <kix@kix.es>
2020-09-03 21:36:09 +02:00
Rodolfo García Peñas (kix)
cefac73af4 six: changed offlineimap/folder/IMAP.py
This patch removes the library six, compatible with python2.

I need change these re-raise calls.

Signed-off-by: Rodolfo García Peñas (kix) <kix@kix.es>
2020-09-03 21:36:01 +02:00
Rodolfo García Peñas (kix)
ba11b2d684 six: changed offlineimap/folder/Maildir.py
This patch removes the library six, compatible with python2.

I need change these re-raise calls.

Signed-off-by: Rodolfo García Peñas (kix) <kix@kix.es>
2020-09-03 21:36:00 +02:00
Rodolfo García Peñas (kix)
4b32d62fe9 six: changed offlineimap/folder/GmailMaildir.py
This patch removes the library six, compatible with python2.

I need change this re-raise call.

Signed-off-by: Rodolfo García Peñas (kix) <kix@kix.es>
2020-09-03 21:35:59 +02:00
Rodolfo García Peñas (kix)
bafcef77fd six: changed offlineimap/folder/Gmail.py
This patch removes the library six, compatible with python2.

I need change this re-raise call.

Signed-off-by: Rodolfo García Peñas (kix) <kix@kix.es>
2020-09-03 21:35:58 +02:00
Rodolfo García Peñas (kix)
75c718c75b six: changed offlineimap/folder/UIDMaps.py
This patch removes the library six, compatible with python2.

I need change these re-raise calls.

Signed-off-by: Rodolfo García Peñas (kix) <kix@kix.es>
2020-09-03 21:35:57 +02:00
Rodolfo García Peñas (kix)
dd3e7616cf six: changed offlineimap/folder/LocalStatusSQLite.py
This patch removes the library six, compatible with python2.

I need change these re-raise calls.

Signed-off-by: Rodolfo García Peñas (kix) <kix@kix.es>
2020-09-03 21:35:56 +02:00
Rodolfo García Peñas (kix)
a922ab3103 six: changed offlineimap/folder/LocalStatus.py
This patch removes the library six, compatible with python2.

I need change these re-raise calls.

Signed-off-by: Rodolfo García Peñas (kix) <kix@kix.es>
2020-09-03 21:35:55 +02:00
Rodolfo García Peñas (kix)
df4b9174d7 IMAP.py __savemessage_fetchheaders decode bytes
This patch changes the function __savemessage_fetchheaders to decode the
bytes retunred by imaplib2.

We need a list of headers, with string values, but imapli2 is providing
a list with bytes. This change convert the values to str.

Signed-off-by: Rodolfo García Peñas (kix) <kix@kix.es>
2020-09-03 21:35:52 +02:00
Rodolfo García Peñas (kix)
b24687fcd6 Gmail.py Get the server response right
Now, the server response is in a list of strings. We need the second
string, so we need read the [1].

Previously, was a list of tuples, so, we used [0][1].

This patch is like the patch IMAP.py Get the server response right, but
now for Gmail.

Signed-off-by: Rodolfo García Peñas (kix) <kix@kix.es>
2020-09-03 21:35:51 +02:00
Rodolfo García Peñas (kix)
8e63f58b22 folder/IMAP.py matching uids is a list
matchinguids variable is a list of UIDs, separated by spaces. You can
check it some lines later, using the split command.

We need decode the bytes value returned by imaplib2 and convert it to
sting.

Signed-off-by: Rodolfo García Peñas (kix) <kix@kix.es>
2020-09-03 21:35:48 +02:00
Rodolfo García Peñas (kix)
41c2ced1d5 IMAP.py _msgs_to_fetch decode bytes
imaplib2 returns the type as string, like "OK" but
returns imapdata as list of bytes, like [b'0'] so we need decode it
to use the existing code
2020-09-01 19:07:52 +02:00
Rodolfo García Peñas (kix)
1ff54e7a7c IMAP.py Get the server response right
Now, the server response is in a list of strings. We need the second
string, so we need read the [1].

Previously, was a list of tuples, so, we used [0][1].
2020-09-01 18:21:27 +02:00