I-Al-Istannen
f0562049b6
Remove Python 3.9 method in crawler
2020-12-30 17:18:04 +01:00
I-Al-Istannen
c978e9edf4
Resolve a few pylint warnings
2020-12-30 14:45:46 +01:00
I-Al-Istannen
2714ac6be6
Send CSRF token to Shibboleth
2020-12-30 14:34:11 +01:00
I-Al-Istannen
9b048a9cfc
Canonize meeting names to a properly formatted date
2020-12-30 14:32:59 +01:00
I-Al-Istannen
f47b137b59
Fix ILIAS init.py and Pferd.py authenticators
2020-12-06 13:15:32 +01:00
Scriptim
83ea15ee83
Use system keyring service for password auth
2020-12-06 13:15:30 +01:00
I-Al-Istannen
ba3c7f85fa
Replace "\" in ILIAS paths as well
...
I am not sure whether anybody really uses a backslash in their names,
but I guess it can't hurt to do this for windows users.
2020-11-19 19:37:28 +01:00
I-Al-Istannen
55e9e719ad
Sanitize "/" in ilias path names
2020-11-12 20:21:24 +01:00
I-Al-Istannen
316b9d7bf4
Prevent too many retries when fetching an ILIAS page
2020-11-04 22:23:56 +01:00
I-Al-Istannen
73c3eb0984
Add option to skip videos in sync_url
2020-10-06 17:20:47 +02:00
I-Al-Istannen
c1ccb6c53e
Allow crawling videos with sync_url
2020-10-06 10:46:06 +02:00
I-Al-Istannen
51a713fa04
Allow crawling courses or folders with sync_url
...
Video folders do not work, if they are passed directly. Their containing
folder must be specified instead.
2020-09-28 20:00:01 +02:00
I-Al-Istannen
e32a49480b
Expose methods to look up course/element names by id / url
2020-09-28 19:16:52 +02:00
I-Al-Istannen
3f0ae729d6
Expand "is course" check to not download magazines or other weird things
2020-09-28 16:43:58 +02:00
I-Al-Istannen
a57ee8b96b
Add timeout to video downloads to work around requests IPv6 bug
2020-08-11 14:40:30 +02:00
Joscha
77a109bb7e
Fix ilias shibboleth authenticator
...
The shibboleth site got a visual overhaul that slightly changed the classes of a
form we need.
2020-07-28 19:13:51 +00:00
I-Al-Istannen
5c4c785e60
Fix HTML file downloading
...
Previously PFERD thought any HTML file was a "Error, no access" page
when downloading. Now it checks whether ILIAS sends a
content-disposition header, telling the browser to download the file. If
that is the case, it was just a HTML file uploaded to ILIAS. If it has
no header, it is probably an error message.
2020-07-15 15:12:14 +02:00
I-Al-Istannen
2aed4f6d1f
Only query the dir_filter for directories
2020-07-13 13:36:12 +02:00
I-Al-Istannen
34152fbe54
Set mtime and atime to ILIAS dates where possible
2020-07-13 13:29:18 +02:00
I-Al-Istannen
23bfa42a0d
Never use the direct download button, as it is currently broken
2020-06-11 13:31:01 +02:00
I-Al-Istannen
8198c9ecaa
Reorder methods a bit
2020-05-30 19:06:36 +02:00
I-Al-Istannen
086b15d10f
Crawl a bit more iteratively
2020-05-30 15:47:15 +02:00
I-Al-Istannen
9d6ce331a5
Use IliasCrawlerEntry entries in the ilias scraper
2020-05-30 15:20:51 +02:00
I-Al-Istannen
821c7ade26
Move video url extraction logic to crawler
2020-05-30 00:22:31 +02:00
I-Al-Istannen
62535b4452
Unpack videos in ILIAS downloader
2020-05-21 22:12:52 +02:00
I-Al-Istannen
c0056e5669
Correctly crawl video pages with multiple pages
2020-05-21 21:38:07 +02:00
I-Al-Istannen
4f56c8f192
Pass element type to ilias directory filter
2020-05-12 14:41:13 +02:00
I-Al-Istannen
a0f9d31d94
Use PrettyLogger warning everywhere
2020-05-10 21:56:12 +02:00
I-Al-Istannen
9850ab1d73
Allow crawling the ILIAS Personal Desktop
2020-05-10 12:16:42 +02:00
I-Al-Istannen
f6faacabb0
Move FatalException to errors.py
2020-05-09 00:11:21 +02:00
I-Al-Istannen
19c1e3ac6f
Fail on invalid ILIAS course ids
2020-05-09 00:11:20 +02:00
I-Al-Istannen
a4c518bf4c
Update date find regex
2020-05-08 22:17:58 +02:00
I-Al-Istannen
057135022f
Try to accept that life sometimes is in English
2020-05-08 22:10:43 +02:00
I-Al-Istannen
755e9aa0d3
Try to add support for Shibboleth TFA token
2020-05-08 21:52:51 +02:00
I-Al-Istannen
56f2394001
Add a download progress bar
2020-05-08 17:09:56 +02:00
I-Al-Istannen
42345ecc61
Demangle "Morgen" too
2020-04-30 12:05:25 +02:00
I-Al-Istannen
920d521d68
Change PrettyLogger.warn to PrettyLogger.warning
2020-04-25 20:11:51 +02:00
I-Al-Istannen
e0b46a306a
Use warn method in IliasCrawler
2020-04-25 20:07:40 +02:00
I-Al-Istannen
8a42a2a396
Move logging into its own file
2020-04-25 20:02:01 +02:00
I-Al-Istannen
80247400a4
Debug log when starting an ilias download
2020-04-25 13:02:07 +02:00
Joscha
292e516297
Change crawler and downloader output
2020-04-24 18:24:44 +00:00
Joscha
5b929f09a2
Move download strategies to downloader
...
Also fixes an issue where the downloader didn't mark files that were not
downloaded due to the strategy used.
2020-04-24 14:27:40 +00:00
I-Al-Istannen
f3f4be2690
More free functions
2020-04-23 19:21:49 +02:00
I-Al-Istannen
076b8c5a1f
Add download strategies to save bandwith
...
Only download files that are newer than the local version.
2020-04-23 18:29:20 +02:00
I-Al-Istannen
13bc78c889
Display reason for ignoring an element in ilias crawler
2020-04-23 13:54:58 +02:00
I-Al-Istannen
dc964a9d98
Remove finished TODOs
2020-04-23 13:30:34 +02:00
I-Al-Istannen
c2b14f3db9
ilias crawler: Use direct download link if possible
2020-04-23 13:08:12 +02:00
Joscha
4b59a7c375
Move around TODOs
2020-04-23 10:49:01 +00:00
I-Al-Istannen
bef210ae77
Rename and implement IliasDirectoryFilter
2020-04-23 12:35:18 +02:00
Joscha
2de4255a78
Add Pferd class
2020-04-23 09:50:32 +00:00
