ffda4e43df
Add extension to link files
2021-05-25 11:41:57 +02:00
69cb2a7734
Add Links option to ilias crawler
...
This allows you to configure what type the link files should have and
whether to create them at all.
2021-05-25 11:41:57 +02:00
85f89a7ff3
Interpret accordions and expandable headers as virtual folders
...
This allows us to find a file named "Test" in an accordion "Acc" as "Acc/Test".
2021-05-24 18:54:26 +02:00
9ce20216b5
Do not set a timeout for whole HTTP request
...
Downloads might take longer!
2021-05-24 18:54:26 +02:00
86ba47541b
Fix cookie loading and saving
2021-05-24 16:55:11 +02:00
492ec6a932
Detect and skip ILIAS tests
2021-05-24 16:36:15 +02:00
342076ee0e
Handle exercise detail containers in ILIAS html parser
2021-05-24 16:22:51 +02:00
d44f6966c2
Log authentication attempts in HTTP crawler
2021-05-24 16:22:11 +02:00
1c1f781be4
Reword some log messages
2021-05-24 13:17:28 +02:00
c687d4a51a
Implement cookie sharing
2021-05-24 13:10:44 +02:00
fca62541ca
De-duplicate element names in ILIAS crawler
...
This prevents any conflicts caused by multiple files with the same name.
Conflicts may still arise due to transforms, but that is out of our
control and a user error.
2021-05-24 00:24:31 +02:00
3ab3581f84
Add timeout for HTTP connection
2021-05-23 23:41:05 +02:00
8dd0689420
Add keyring authentication to ILIAS CLI
2021-05-23 23:04:18 +02:00
79be6e1dc5
Switch some other options to BooleanOptionalAction
2021-05-23 22:49:09 +02:00
edbd92dbbf
Add --status and --report flags
2021-05-23 22:41:59 +02:00
27b5a8e490
Rename log.action to log.status
2021-05-23 22:40:33 +02:00
1f400d5964
Implement BooleanOptionalAction
2021-05-23 22:26:59 +02:00
0ca0680165
Simplify --version
2021-05-23 21:40:48 +02:00
ce1dbda5b4
Overhaul colours
...
"Crawled" and "Downloaded" are now printed less bright than "Crawling" and
"Downloading" as they're not as important. Explain topics are printed in yellow
to stand out a bit more from the cyan action messages.
2021-05-23 21:33:04 +02:00
9cce78669f
Print report after all crawlers have finished
2021-05-23 21:17:13 +02:00
6ca0ecdf05
Load and store reports
2021-05-23 20:46:29 +02:00
6e9f8fd391
Add a keyring authenticator
2021-05-23 19:44:12 +02:00
2fdf24495b
Restructure crawling and auth related modules
2021-05-23 19:16:42 +02:00
bbf9f8f130
Add -C as alias for --crawler
2021-05-23 19:06:09 +02:00
37f8d84a9c
Output total amount of http requests in HTTP Crawler
2021-05-23 19:00:01 +02:00
5edd868d5b
Fix always-smart redownloading the wrong files
2021-05-23 18:49:34 +02:00
e4e5e83be6
Fix downloader using crawl bar
...
Looks like I made a dumb copy-paste error. Now the download bar shows the proper
progress and speed again.
2021-05-23 18:39:43 +02:00
74c7b39dc8
Clean up files in alphabetical order
2021-05-23 18:39:25 +02:00
445dffc987
Reword some explanations
2021-05-23 18:35:32 +02:00
d97d6bf147
Fix handling nested ILIAS folders
2021-05-23 18:29:28 +02:00
79efdb56f7
Adjust ILIAS html explain messages
2021-05-23 18:24:25 +02:00
a9af56a5e9
Improve specifying crawlers via CLI
...
Instead of removing the sections of unselected crawlers from the config file,
crawler selection now happens in the Pferd after loading the crawlers and is
more sophisticated. It also has better error messages.
2021-05-23 18:18:50 +02:00
59f13bb8d6
Explain ILIAS HTML parsing and add some warnings
2021-05-23 18:14:54 +02:00
463f8830d7
Add warn_contd
2021-05-23 18:14:54 +02:00
05ad06fbc1
Only enclose get_page in iorepeat in ILIAS crawler
...
We previously also gathered in there, which could lead to some more
surprises when the method was retried.
2021-05-23 18:14:51 +02:00
29d5a40c57
Replace asyncio.gather with custom Crawler function
2021-05-23 17:25:16 +02:00
c0cecf8363
Log crawl and download actions more extensively
2021-05-23 16:25:44 +02:00
b998339002
Fix cleanup logging of paths
2021-05-23 16:25:44 +02:00
245c9c3dcc
Explain output dir decisions and steps
2021-05-23 16:25:44 +02:00
d8f26a789e
Implement CLI Command for ilias crawler
2021-05-23 13:30:42 +02:00
e1d18708b3
Rename "no_videos" to videos
2021-05-23 13:30:42 +02:00
b44b49476d
Fix noncritical and anoncritical decorators
...
I must've forgot to update the anoncritical decorator when I last changed the
noncritical decorator. Also, every exception should make the crawler not
error_free, not just CrawlErrors.
2021-05-23 13:24:53 +02:00
7e0bb06259
Clean up TODOs
2021-05-23 12:47:30 +02:00
ecdedfa1cf
Add no-videos flag to ILIAS crawler
2021-05-23 12:37:01 +02:00
3d4b997d4a
Retry crawl_url and work around Python's closure handling
...
Closures capture the scope and not the variables. Therefore, any
type-narrowing performed by mypy on captured variables is lost inside
the closure.
2021-05-23 12:28:15 +02:00
e81005ae4b
Fix CLI arguments
2021-05-23 12:24:21 +02:00
33a81a5f5c
Document authentication in HTTP crawler and rename prepare_request
2021-05-23 11:55:34 +02:00
25e2abdb03
Improve transformer explain wording
2021-05-23 11:45:14 +02:00
803e5628a2
Clean up logging
...
Paths are now (hopefully) logged consistently across all crawlers
2021-05-23 11:37:19 +02:00
c88f20859a
Explain config file dumping
2021-05-23 11:04:50 +02:00
