mirrors/pferd
1
0
Fork 0
mirror of https://github.com/Garmelon/PFERD.git synced 2023-12-21 10:23:01 +01:00
Code Issues Packages Projects Releases Wiki Activity
670 Commits 6 Branches 43 Tags 1.6 MiB
533bc27439
Commit Graph

53 Commits

Author SHA1 Message Date
I-Al-Istannen
50b50513c6 Ignore SCORM learning modules 2023-08-29 13:51:19 +02:00
I-Al-Istannen
87b67e9271 Crawl files in the info tab 2023-08-29 12:41:15 +02:00
I-Al-Istannen
2184ac8040 Add support for ILIAS mediacast listings 2023-08-27 11:43:05 +02:00
I-Al-Istannen
68c398f1fe Add support for ILIAS learning modules 2023-08-02 13:34:54 +02:00
I-Al-Istannen
d204dac8ce Detect unexpected root page redirects and abort operation 2023-07-29 18:36:33 +02:00
I-Al-Istannen
c020cccc64 Include found paths in "second path found" warning 2022-10-29 14:08:29 +02:00
I-Al-Istannen
1b6be6bd79 Handle content pages in cards 2022-10-24 18:37:26 +02:00
I-Al-Istannen
e1430e6298 Handle (and ignore) surveys 2022-10-24 18:37:26 +02:00
I-Al-Istannen
5fdd40204b Unwrap future meetings when ILIAS hides them behind a pagination 2022-10-24 14:33:58 +02:00
I-Al-Istannen
fb4631ba18 Fix ilias background login 2022-10-24 13:13:36 +02:00
I-Al-Istannen
d72fc2760b Handle empty forums 2022-10-24 13:12:17 +02:00
I-Al-Istannen
d9b111cec2 Correctly nest description entries 2022-08-14 21:45:33 +02:00
I-Al-Istannen
345f52a1f6 Detect new login button 2022-08-14 21:41:29 +02:00
I-Al-Istannen
46fb782798 Add forum crawling 
This downloads all forum posts when needed and saves each thread in its
own html file, named after the thread title.
 2022-05-24 23:43:53 +02:00
I-Al-Istannen
846c29aee1 Download page descriptions 2022-05-11 21:16:56 +02:00
Joscha
602044ff1b Fix mypy errors and add missing await 2022-04-27 22:52:50 +02:00
I-Al-Istannen
a2831fbea2 Fix shib authentication 
Authentication failed previously if the shib session was still valid.
If Shibboleth gets a request and the session is still valid, it directly
responds without a second redirect.
 2022-04-27 13:55:24 +02:00
I-Al-Istannen
86e2e226dc Notify user when shibboleth presents new entitlements 2022-04-03 11:37:08 +02:00
Joscha
4f022e2d19 Reword changelog 2022-01-15 15:06:02 +01:00
I-Al-Istannen
f47e7374d2 Use fixed windows path for video cache 2022-01-15 12:00:30 +01:00
I-Al-Istannen
57ec51e95a Fix login after shib url parser change 2022-01-14 20:17:27 +01:00
I-Al-Istannen
d30f25ee97 Detect shib login page as login page 
And do not assume we are logged in...
 2022-01-08 20:28:45 +01:00
I-Al-Istannen
10d9d74528 Bail out when crawling recursive courses 2022-01-08 20:28:30 +01:00
I-Al-Istannen
e32c1f000f Fix mtime for single streams 2022-01-08 18:05:48 +01:00
I-Al-Istannen
6f3cfd4396 Fix personal desktop crawling 2022-01-08 16:58:15 +01:00
I-Al-Istannen
462d993fbc Fix local video path cache (hopefully) 2022-01-08 00:27:48 +01:00
I-Al-Istannen
eac2e34161 Fix is_logged_in for ILIAS 7 2022-01-07 23:32:31 +01:00
I-Al-Istannen
90cb6e989b Do not download single videos if cache does not exist 2021-11-06 23:21:15 +01:00
I-Al-Istannen
6289938d7c Do not stop crawling files when encountering a CrawlWarning 2021-11-06 12:09:51 +01:00
I-Al-Istannen
f9a3f9b9f2 Handle multi-stream videos 2021-10-30 18:12:29 +02:00
I-Al-Istannen
ee67f9f472 Sort elements by ILIAS id to ensure deterministic ordering 2021-07-06 17:45:48 +02:00
I-Al-Istannen
8ec3f41251 Crawl ilias booking objects as links 2021-07-06 16:15:25 +02:00
I-Al-Istannen
89be07d4d3 Use final crawl path in HTML parsing message 2021-07-03 17:05:48 +02:00
I-Al-Istannen
91200f3684 Fix nondeterministic name deduplication 2021-07-03 12:09:55 +02:00
I-Al-Istannen
6e4d423c81 Crawl all video stages in one crawl bar 
This ensures folders are not renamed, as they are crawled twice
 2021-06-13 17:18:45 +02:00
I-Al-Istannen
70ec64a48b Fix wrong base URL for multi-stage pages 2021-06-13 15:44:47 +02:00
Joscha
f40820c41f Warn if using concurrent tasks with kit-ilias-web 2021-05-31 20:18:20 +00:00
I-Al-Istannen
1ca6740e05 Improve log messages when parsing ILIAS HTML 
Previously some logs were split around an "await", which isn't a great
idea.
 2021-05-27 17:59:22 +02:00
I-Al-Istannen
5beb4d9a2d Fix renaming conflict with multi-stage video elements 2021-05-27 15:41:00 +02:00
I-Al-Istannen
19eed5bdff Fix authentication logic conflicts with videos 2021-05-27 15:41:00 +02:00
I-Al-Istannen
2d8dcc87ff Send CSRF token in TFA request 2021-05-25 22:50:40 +02:00
I-Al-Istannen
66f0e398a1 Await result in tfa authenticate path 2021-05-25 19:19:51 +02:00
I-Al-Istannen
486699cef3 Create anonymous TFA authenticator in ilias crawler 
This ensures that *some* TFA authenticator is always present when
authenticating, even if none is specified in the config.

The TfaAuthenticator does not depend on any configured values, so it can
be created on-demand.
 2021-05-25 15:11:52 +02:00
Joscha
61430c8739 Overhaul config and CLI option names 2021-05-25 14:23:38 +02:00
I-Al-Istannen
651b087932 Use cl/dl deduplication mechanism for ILIAS crawler 2021-05-25 12:15:38 +02:00
I-Al-Istannen
ffda4e43df Add extension to link files 2021-05-25 11:41:57 +02:00
I-Al-Istannen
69cb2a7734 Add Links option to ilias crawler 
This allows you to configure what type the link files should have and
whether to create them at all.
 2021-05-25 11:41:57 +02:00
I-Al-Istannen
492ec6a932 Detect and skip ILIAS tests 2021-05-24 16:36:15 +02:00
I-Al-Istannen
342076ee0e Handle exercise detail containers in ILIAS html parser 2021-05-24 16:22:51 +02:00
Joscha
c687d4a51a Implement cookie sharing 2021-05-24 13:10:44 +02:00