mirrors/pferd
1
0
Fork 0
mirror of https://github.com/Garmelon/PFERD.git synced 2023-12-21 10:23:01 +01:00
Code Issues Packages Projects Releases Wiki Activity
655 Commits 6 Branches 43 Tags 1.6 MiB
d512fc24c5
Commit Graph

61 Commits

Author SHA1 Message Date
Joscha
6d44aac278 Bump version to 3.4.3 2022-11-29 18:22:19 +01:00
c0derMo
55a2de6b88 Fix crawling English opencast 2022-11-29 18:13:56 +01:00
Joscha
c0d6d8b229 Use url after redirect for relative links 2022-11-21 18:10:45 +01:00
Joscha
07200bbde5 Document ilias web crawler's forums option 2022-10-31 14:12:27 +01:00
I-Al-Istannen
c020cccc64 Include found paths in "second path found" warning 2022-10-29 14:08:29 +02:00
Joscha
259cfc20cc Bump version to 3.4.2 2022-10-26 18:26:17 +02:00
Joscha
37b51a66d8 Update changelog 2022-10-26 18:22:37 +02:00
I-Al-Istannen
f47d2f11d8 Append trailing slash to kit-ipd links to ensure urljoin works as expected 2022-10-25 20:28:22 +02:00
I-Al-Istannen
d72fc2760b Handle empty forums 2022-10-24 13:12:17 +02:00
I-Al-Istannen
4a51aaa4f5 Fix forum crawling crashing for empty threads 2022-10-19 22:59:33 +02:00
Joscha
66a5b1ba02 Bump version to 3.4.1 2022-08-17 13:24:01 +02:00
I-Al-Istannen
aa5a3a10bc Adjust changelog 2022-08-14 21:48:59 +02:00
Joscha
ed24366aba Add pass authenticator 2022-06-05 10:04:42 +02:00
I-Al-Istannen
46fb782798 Add forum crawling 
This downloads all forum posts when needed and saves each thread in its
own html file, named after the thread title.
 2022-05-24 23:43:53 +02:00
I-Al-Istannen
846c29aee1 Download page descriptions 2022-05-11 21:16:56 +02:00
Joscha
616b0480f7 Simplify IPD crawler link regex 2022-05-08 18:18:05 +02:00
I-Al-Istannen
2f0e04ce13 Adjust changelog 2022-05-05 22:57:55 +02:00
Joscha
af2cc1169a Mention href for users of link_regex option 2022-05-05 14:36:03 +02:00
Joscha
bc3fa36637 Fix IPD crawler crashing on weird HTML comments 2022-05-05 14:35:42 +02:00
Joscha
afbd03f777 Fix docs 2022-05-05 14:35:42 +02:00
I-Al-Istannen
b8fe25c580 Add .cpp to ipd link regex 2022-05-04 14:19:26 +02:00
Joscha
a241672726 Bump version to 3.4.0 2022-05-01 22:29:06 +02:00
Joscha
31631fb409 Increase minimum python version to 3.9 2022-04-27 22:52:50 +02:00
I-Al-Istannen
00db348218 Update changelog 2022-04-27 22:03:52 +02:00
Joscha
ba3d299c05 Fix changelog 2022-04-27 21:26:24 +02:00
Joscha
07a21f80a6 Link to unofficial packages 2022-04-27 21:15:33 +02:00
I-Al-Istannen
f17b9b68f4 Add shibboleth authentication fix to changelog 2022-04-27 14:01:40 +02:00
I-Al-Istannen
86e2e226dc Notify user when shibboleth presents new entitlements 2022-04-03 11:37:08 +02:00
Joscha
86947e4874 Bump version to 3.3.1 2022-01-15 15:11:22 +01:00
Joscha
4f022e2d19 Reword changelog 2022-01-15 15:06:02 +01:00
I-Al-Istannen
f47e7374d2 Use fixed windows path for video cache 2022-01-15 12:00:30 +01:00
I-Al-Istannen
57ec51e95a Fix login after shib url parser change 2022-01-14 20:17:27 +01:00
Joscha
0045124a4e Bump version to 3.3.0 2022-01-09 21:09:09 +01:00
I-Al-Istannen
9618aae83b Add content pages to changelog 2022-01-09 18:32:58 +01:00
I-Al-Istannen
e9d2d05030 Update changelog 2022-01-09 11:48:26 +01:00
I-Al-Istannen
ad3f4955f7 Update changelog 2021-10-30 18:14:39 +02:00
lukasprobst
55ea304ff3 Disable interpolation of ConfigParser 2021-10-25 23:37:42 +02:00
Joscha
fee12b3d9e Fix changelog 2021-10-25 17:44:12 +00:00
I-Al-Istannen
6673077397 Add kit-ipd crawler 2021-10-21 13:20:21 +02:00
Joscha
742632ed8d Bump version to 3.2.0 2021-08-04 18:27:26 +00:00
Joscha
544d45cbc5 Catch non-critical exceptions at crawler top level 2021-07-13 15:42:11 +02:00
Joscha
86f79ff1f1 Update changelog 2021-07-07 15:23:58 +02:00
Joscha
75fde870c2 Bump version to 3.1.0 2021-06-13 17:23:18 +02:00
I-Al-Istannen
70ec64a48b Fix wrong base URL for multi-stage pages 2021-06-13 15:44:47 +02:00
Joscha
70b33ecfd9 Add migration notes to changelog 
Also clean up some other formatting for consistency
 2021-06-13 15:06:50 +02:00
Joscha
61d902d715 Overhaul transform logic 
-re-> arrows now rename their parent directories (like -->) and don't require a
full match (like -exact->). Their old behaviour is available as -exact-re->.

Also, this change adds the ">>" arrow head, which modifies the current path and
continues to the next rule when it matches.
 2021-06-09 22:45:52 +02:00
I-Al-Istannen
8ab462fb87 Use the exercise label instead of the button name as path 2021-06-04 19:24:23 +02:00
Joscha
df3ad3d890 Add 'skip' option to crawlers 2021-06-04 18:47:13 +02:00
Joscha
fc31100a0f Always use '/' as path separator for regex rules 
Previously, regex-matching paths on windows would, in some cases, require four
backslashes ('\\\\') to escape a single path separator. That's just too much.

With this commit, regex transforms now use '/' instead of '\' as path separator,
meaning rules can more easily be shared between platforms (although they are not
guaranteed to be 100% compatible since on Windows, '\' is still recognized as a
path separator).

To make rules more intuitive to write, local relative paths are now also printed
with '/' as path separator on Windows. Since Windows also accepts '/' as path
separator, this change doesn't really affect other rules that parse their sides
as paths.
 2021-06-04 18:12:45 +02:00
Joscha
85b9f45085 Bump version to 3.0.1 2021-06-01 09:49:30 +00:00