I-Al-Istannen
10e1a5e871
De-Async ilias crawler
2023-04-19 10:12:48 +02:00
I-Al-Istannen
c020cccc64
Include found paths in "second path found" warning
2022-10-29 14:08:29 +02:00
I-Al-Istannen
1b6be6bd79
Handle content pages in cards
2022-10-24 18:37:26 +02:00
I-Al-Istannen
e1430e6298
Handle (and ignore) surveys
2022-10-24 18:37:26 +02:00
I-Al-Istannen
5fdd40204b
Unwrap future meetings when ILIAS hides them behind a pagination
2022-10-24 14:33:58 +02:00
I-Al-Istannen
fb4631ba18
Fix ilias background login
2022-10-24 13:13:36 +02:00
I-Al-Istannen
d72fc2760b
Handle empty forums
2022-10-24 13:12:17 +02:00
I-Al-Istannen
d9b111cec2
Correctly nest description entries
2022-08-14 21:45:33 +02:00
I-Al-Istannen
345f52a1f6
Detect new login button
2022-08-14 21:41:29 +02:00
I-Al-Istannen
46fb782798
Add forum crawling
...
This downloads all forum posts when needed and saves each thread in its
own html file, named after the thread title.
2022-05-24 23:43:53 +02:00
I-Al-Istannen
846c29aee1
Download page descriptions
2022-05-11 21:16:56 +02:00
Joscha
602044ff1b
Fix mypy errors and add missing await
2022-04-27 22:52:50 +02:00
I-Al-Istannen
a2831fbea2
Fix shib authentication
...
Authentication failed previously if the shib session was still valid.
If Shibboleth gets a request and the session is still valid, it directly
responds without a second redirect.
2022-04-27 13:55:24 +02:00
I-Al-Istannen
86e2e226dc
Notify user when shibboleth presents new entitlements
2022-04-03 11:37:08 +02:00
Joscha
4f022e2d19
Reword changelog
2022-01-15 15:06:02 +01:00
I-Al-Istannen
f47e7374d2
Use fixed windows path for video cache
2022-01-15 12:00:30 +01:00
I-Al-Istannen
57ec51e95a
Fix login after shib url parser change
2022-01-14 20:17:27 +01:00
I-Al-Istannen
d30f25ee97
Detect shib login page as login page
...
And do not assume we are logged in...
2022-01-08 20:28:45 +01:00
I-Al-Istannen
10d9d74528
Bail out when crawling recursive courses
2022-01-08 20:28:30 +01:00
I-Al-Istannen
e32c1f000f
Fix mtime for single streams
2022-01-08 18:05:48 +01:00
I-Al-Istannen
6f3cfd4396
Fix personal desktop crawling
2022-01-08 16:58:15 +01:00
I-Al-Istannen
462d993fbc
Fix local video path cache (hopefully)
2022-01-08 00:27:48 +01:00
I-Al-Istannen
eac2e34161
Fix is_logged_in for ILIAS 7
2022-01-07 23:32:31 +01:00
I-Al-Istannen
90cb6e989b
Do not download single videos if cache does not exist
2021-11-06 23:21:15 +01:00
I-Al-Istannen
6289938d7c
Do not stop crawling files when encountering a CrawlWarning
2021-11-06 12:09:51 +01:00
I-Al-Istannen
f9a3f9b9f2
Handle multi-stream videos
2021-10-30 18:12:29 +02:00
I-Al-Istannen
ee67f9f472
Sort elements by ILIAS id to ensure deterministic ordering
2021-07-06 17:45:48 +02:00
I-Al-Istannen
8ec3f41251
Crawl ilias booking objects as links
2021-07-06 16:15:25 +02:00
I-Al-Istannen
89be07d4d3
Use final crawl path in HTML parsing message
2021-07-03 17:05:48 +02:00
I-Al-Istannen
91200f3684
Fix nondeterministic name deduplication
2021-07-03 12:09:55 +02:00
I-Al-Istannen
6e4d423c81
Crawl all video stages in one crawl bar
...
This ensures folders are not renamed, as they are crawled twice
2021-06-13 17:18:45 +02:00
I-Al-Istannen
70ec64a48b
Fix wrong base URL for multi-stage pages
2021-06-13 15:44:47 +02:00
Joscha
f40820c41f
Warn if using concurrent tasks with kit-ilias-web
2021-05-31 20:18:20 +00:00
I-Al-Istannen
1ca6740e05
Improve log messages when parsing ILIAS HTML
...
Previously some logs were split around an "await", which isn't a great
idea.
2021-05-27 17:59:22 +02:00
I-Al-Istannen
5beb4d9a2d
Fix renaming conflict with multi-stage video elements
2021-05-27 15:41:00 +02:00
I-Al-Istannen
19eed5bdff
Fix authentication logic conflicts with videos
2021-05-27 15:41:00 +02:00
I-Al-Istannen
2d8dcc87ff
Send CSRF token in TFA request
2021-05-25 22:50:40 +02:00
I-Al-Istannen
66f0e398a1
Await result in tfa authenticate path
2021-05-25 19:19:51 +02:00
I-Al-Istannen
486699cef3
Create anonymous TFA authenticator in ilias crawler
...
This ensures that *some* TFA authenticator is always present when
authenticating, even if none is specified in the config.
The TfaAuthenticator does not depend on any configured values, so it can
be created on-demand.
2021-05-25 15:11:52 +02:00
Joscha
61430c8739
Overhaul config and CLI option names
2021-05-25 14:23:38 +02:00
I-Al-Istannen
651b087932
Use cl/dl deduplication mechanism for ILIAS crawler
2021-05-25 12:15:38 +02:00
I-Al-Istannen
ffda4e43df
Add extension to link files
2021-05-25 11:41:57 +02:00
I-Al-Istannen
69cb2a7734
Add Links option to ilias crawler
...
This allows you to configure what type the link files should have and
whether to create them at all.
2021-05-25 11:41:57 +02:00
I-Al-Istannen
492ec6a932
Detect and skip ILIAS tests
2021-05-24 16:36:15 +02:00
I-Al-Istannen
342076ee0e
Handle exercise detail containers in ILIAS html parser
2021-05-24 16:22:51 +02:00
Joscha
c687d4a51a
Implement cookie sharing
2021-05-24 13:10:44 +02:00
I-Al-Istannen
fca62541ca
De-duplicate element names in ILIAS crawler
...
This prevents any conflicts caused by multiple files with the same name.
Conflicts may still arise due to transforms, but that is out of our
control and a user error.
2021-05-24 00:24:31 +02:00
I-Al-Istannen
3ab3581f84
Add timeout for HTTP connection
2021-05-23 23:41:05 +02:00
Joscha
2fdf24495b
Restructure crawling and auth related modules
2021-05-23 19:16:42 +02:00
