Joscha
df3ad3d890
Add 'skip' option to crawlers
2021-06-04 18:47:13 +02:00
Joscha
533f75ea71
Add --debug-transforms flag
2021-05-26 11:37:32 +02:00
Joscha
adb5d4ade3
Print files that are *not* deleted by cleanup
...
These are files that are not present on the remote source any more, but still
present locally. They also show up in the report.
2021-05-26 10:58:19 +02:00
Joscha
5a331663e4
Rename functions for consistency
2021-05-25 15:49:06 +02:00
Joscha
f68849c65f
Fix rules not being parsed entirely
2021-05-25 15:42:46 +02:00
Joscha
edb52a989e
Print report even if exiting due to Ctrl+C
2021-05-25 15:35:36 +02:00
Joscha
22c2259adb
Clean up authenticator exceptions
...
- Renamed to *Error for consistency
- Treating AuthError like CrawlError
2021-05-25 14:23:38 +02:00
Joscha
1c1f781be4
Reword some log messages
2021-05-24 13:17:28 +02:00
Joscha
c687d4a51a
Implement cookie sharing
2021-05-24 13:10:44 +02:00
Joscha
9cce78669f
Print report after all crawlers have finished
2021-05-23 21:17:13 +02:00
Joscha
2fdf24495b
Restructure crawling and auth related modules
2021-05-23 19:16:42 +02:00
Joscha
a9af56a5e9
Improve specifying crawlers via CLI
...
Instead of removing the sections of unselected crawlers from the config file,
crawler selection now happens in the Pferd after loading the crawlers and is
more sophisticated. It also has better error messages.
2021-05-23 18:18:50 +02:00
Joscha
662191eca9
Fix crash as soon as first cl or dl token was acquired
2021-05-22 20:25:58 +00:00
Joscha
9889ce6b57
Improve PFERD error handling
2021-05-22 21:13:25 +02:00
Joscha
4b68fa771f
Move logging logic to singleton
...
- Renamed module and class because "conductor" didn't make a lot of sense
- Used singleton approach (there's only one stdout after all)
- Redesigned progress bars (now with download speed!)
2021-05-18 22:45:19 +02:00
Joscha
595de88d96
Fix authenticator and crawler names
...
Now, the "auth:" and "crawl:" parts are considered part of the name. This fixes
crawlers not being able to find their authenticators.
2021-05-15 15:25:05 +02:00
Joscha
0acdee15a0
Let crawlers obtain authenticators
2021-05-13 18:57:20 +02:00
Joscha
0459ed093e
Add simple authenticator
...
... including some required authenticator infrastructure
2021-05-11 00:28:03 +02:00
Joscha
d5f29f01c5
Use global conductor instance
...
The switch from crawler-local conductors to a single pferd-global conductor was
made to prepare for auth section credential providers.
2021-05-11 00:05:04 +02:00
Joscha
60cd9873bc
Add local file crawler
2021-05-06 01:02:40 +02:00
Joscha
bbfdadc463
Implement output directory
2021-05-05 18:08:34 +02:00
Joscha
91c33596da
Load crawlers from config file
2021-04-30 16:22:14 +02:00
Joscha
3ea86d18a0
Jerry-rig DummyCrawler to run
2021-04-29 13:45:04 +02:00
I-Al-Istannen
f47b137b59
Fix ILIAS init.py and Pferd.py authenticators
2020-12-06 13:15:32 +01:00
I-Al-Istannen
9f6dc56a7b
Use a strategy to decide conflict resolution
2020-12-02 19:32:57 +01:00
Christophe
f3a4663491
Add passive/no_prompt flag
2020-12-02 18:24:07 +01:00
I-Al-Istannen
ef343dec7c
Merge organizer download summaries
2020-11-04 15:06:58 +01:00
I-Al-Istannen
f4abe3197c
Add ipd crawler
2020-11-03 21:15:40 +01:00
I-Al-Istannen
51a713fa04
Allow crawling courses or folders with sync_url
...
Video folders do not work, if they are passed directly. Their containing
folder must be specified instead.
2020-09-28 20:00:01 +02:00
I-Al-Istannen
a57ee8b96b
Add timeout to video downloads to work around requests IPv6 bug
2020-08-11 14:40:30 +02:00
I-Al-Istannen
77874b432b
Also add personal_desktop to download summary
2020-07-15 22:47:44 +02:00
I-Al-Istannen
d9ea688145
Use pretty logger for summaries
2020-06-26 17:24:36 +02:00
Joscha
402ae81335
Fix type hints
2020-06-26 13:17:44 +00:00
Daniel Augustin
739522a151
Move download summary into a separate class
2020-06-25 23:07:11 +02:00
Daniel Augustin
6c034209b6
Add deleted files to summary
2020-06-25 22:00:28 +02:00
Daniel Augustin
f6fbd5e4bb
Add download summary
2020-06-25 19:19:34 +02:00
I-Al-Istannen
b969a1854a
Remove unneeded whitespace
2020-05-30 00:22:31 +02:00
Joscha
072c6630bf
Avoid logging import in config
2020-05-12 18:19:23 +00:00
I-Al-Istannen
4f56c8f192
Pass element type to ilias directory filter
2020-05-12 14:41:13 +02:00
I-Al-Istannen
9850ab1d73
Allow crawling the ILIAS Personal Desktop
2020-05-10 12:16:42 +02:00
I-Al-Istannen
9950144e97
Allow passing a playlist URL to diva instead of an id
2020-05-10 11:17:13 +02:00
I-Al-Istannen
f6faacabb0
Move FatalException to errors.py
2020-05-09 00:11:21 +02:00
I-Al-Istannen
19c1e3ac6f
Fail on invalid ILIAS course ids
2020-05-09 00:11:20 +02:00
I-Al-Istannen
afa48c2d2d
Swallow and print errors instead of crashing
2020-05-09 00:10:54 +02:00
I-Al-Istannen
bee3d70998
Added a diva playlist downloader
2020-04-30 17:18:45 +02:00
I-Al-Istannen
8a42a2a396
Move logging into its own file
2020-04-25 20:02:01 +02:00
Joscha
1aaa6e7ab5
Use PathLike everywhere
2020-04-24 18:41:14 +00:00
Joscha
7f53543324
Satisfy pylint and add todo
2020-04-24 18:26:28 +00:00
Joscha
8258fa8919
Add test run option to PFERD
2020-04-24 18:00:21 +00:00
Joscha
5b929f09a2
Move download strategies to downloader
...
Also fixes an issue where the downloader didn't mark files that were not
downloaded due to the strategy used.
2020-04-24 14:27:40 +00:00
