ba3c7f85fa
Replace "\" in ILIAS paths as well
...
I am not sure whether anybody really uses a backslash in their names,
but I guess it can't hurt to do this for windows users.
2020-11-19 19:37:28 +01:00
8ebf0eab16
Sort download summary
2020-11-17 21:36:04 +01:00
cd90a60dee
Move "sanitize_windows_path" to PFERD.transform
2020-11-12 20:52:46 +01:00
55e9e719ad
Sanitize "/" in ilias path names
2020-11-12 20:21:24 +01:00
316b9d7bf4
Prevent too many retries when fetching an ILIAS page
2020-11-04 22:23:56 +01:00
f830b42a36
Fix duplicate files in download summary
2020-11-04 21:49:35 +01:00
ef343dec7c
Merge organizer download summaries
2020-11-04 15:06:58 +01:00
0da2fafcd8
Fix links outside tables
2020-11-04 14:46:15 +01:00
f4abe3197c
Add ipd crawler
2020-11-03 21:15:40 +01:00
38d4f5b4c9
Do not fail only empty courses
2020-11-03 20:09:54 +01:00
73c3eb0984
Add option to skip videos in sync_url
2020-10-06 17:20:47 +02:00
c1ccb6c53e
Allow crawling videos with sync_url
2020-10-06 10:46:06 +02:00
51a713fa04
Allow crawling courses or folders with sync_url
...
Video folders do not work, if they are passed directly. Their containing
folder must be specified instead.
2020-09-28 20:00:01 +02:00
e32a49480b
Expose methods to look up course/element names by id / url
2020-09-28 19:16:52 +02:00
3f0ae729d6
Expand "is course" check to not download magazines or other weird things
2020-09-28 16:43:58 +02:00
55678d7fee
Pass string down to FileCookieJar
...
Some python versions just can't handle it *despite the documentation
stating they should*.
2020-08-12 09:09:14 +02:00
a57ee8b96b
Add timeout to video downloads to work around requests IPv6 bug
2020-08-11 14:40:30 +02:00
77a109bb7e
Fix ilias shibboleth authenticator
...
The shibboleth site got a visual overhaul that slightly changed the classes of a
form we need.
2020-07-28 19:13:51 +00:00
a3e1864a26
Allow long paths on windows
...
If you start PFERD a few folders deep in your home directory, it is
quite easy to reach the maximum path length limit on Windows (260
chars). This patch opts in to long paths ("\\?\" prefix) which lift that
restriction at the cost of ugly path names.
2020-07-25 13:44:49 +02:00
77874b432b
Also add personal_desktop to download summary
2020-07-15 22:47:44 +02:00
5c4c785e60
Fix HTML file downloading
...
Previously PFERD thought any HTML file was a "Error, no access" page
when downloading. Now it checks whether ILIAS sends a
content-disposition header, telling the browser to download the file. If
that is the case, it was just a HTML file uploaded to ILIAS. If it has
no header, it is probably an error message.
2020-07-15 15:12:14 +02:00
2aed4f6d1f
Only query the dir_filter for directories
2020-07-13 13:36:12 +02:00
34152fbe54
Set mtime and atime to ILIAS dates where possible
2020-07-13 13:29:18 +02:00
c26c9352f1
Make DownloadSummary private, provide property accessors
2020-06-26 17:30:45 +02:00
d9ea688145
Use pretty logger for summaries
2020-06-26 17:24:36 +02:00
e4b1fac045
Satisfy pylint
2020-06-26 15:38:22 +02:00
402ae81335
Fix type hints
2020-06-26 13:17:44 +00:00
52f31e2783
Add type hints to DownloadSummary
2020-06-26 13:02:37 +02:00
739522a151
Move download summary into a separate class
2020-06-25 23:07:11 +02:00
6c034209b6
Add deleted files to summary
2020-06-25 22:00:28 +02:00
f6fbd5e4bb
Add download summary
2020-06-25 19:19:34 +02:00
7024db1f13
Use transient progessbar
...
This will ensure no pesky newline ends up in the output, even on
windows.
2020-06-25 18:03:12 +02:00
23bfa42a0d
Never use the direct download button, as it is currently broken
2020-06-11 13:31:01 +02:00
fdb57884ed
Touch files with same content to update timestamps
2020-05-31 20:27:15 +02:00
8198c9ecaa
Reorder methods a bit
2020-05-30 19:06:36 +02:00
086b15d10f
Crawl a bit more iteratively
2020-05-30 15:47:15 +02:00
9d6ce331a5
Use IliasCrawlerEntry entries in the ilias scraper
2020-05-30 15:20:51 +02:00
821c7ade26
Move video url extraction logic to crawler
2020-05-30 00:22:31 +02:00
b969a1854a
Remove unneeded whitespace
2020-05-30 00:22:31 +02:00
62535b4452
Unpack videos in ILIAS downloader
2020-05-21 22:12:52 +02:00
c0056e5669
Correctly crawl video pages with multiple pages
2020-05-21 21:38:07 +02:00
03a801eecc
Correctly type hint swallow_and_print_errors decorator
2020-05-12 21:03:53 +02:00
072c6630bf
Avoid logging import in config
2020-05-12 18:19:23 +00:00
4f56c8f192
Pass element type to ilias directory filter
2020-05-12 14:41:13 +02:00
4fdb67128d
Fetch correct diva playlist id
2020-05-11 00:25:34 +02:00
a0f9d31d94
Use PrettyLogger warning everywhere
2020-05-10 21:56:12 +02:00
e7b08420ba
Warn when a marked file is added again
2020-05-10 21:42:30 +02:00
c1b21f7772
Only remove a progress task when we added it
2020-05-10 12:28:30 +02:00
9850ab1d73
Allow crawling the ILIAS Personal Desktop
2020-05-10 12:16:42 +02:00
9950144e97
Allow passing a playlist URL to diva instead of an id
2020-05-10 11:17:13 +02:00
