c687d4a51a
Implement cookie sharing
2021-05-24 13:10:44 +02:00
9cce78669f
Print report after all crawlers have finished
2021-05-23 21:17:13 +02:00
2fdf24495b
Restructure crawling and auth related modules
2021-05-23 19:16:42 +02:00
a9af56a5e9
Improve specifying crawlers via CLI
...
Instead of removing the sections of unselected crawlers from the config file,
crawler selection now happens in the Pferd after loading the crawlers and is
more sophisticated. It also has better error messages.
2021-05-23 18:18:50 +02:00
662191eca9
Fix crash as soon as first cl or dl token was acquired
2021-05-22 20:25:58 +00:00
9889ce6b57
Improve PFERD error handling
2021-05-22 21:13:25 +02:00
4b68fa771f
Move logging logic to singleton
...
- Renamed module and class because "conductor" didn't make a lot of sense
- Used singleton approach (there's only one stdout after all)
- Redesigned progress bars (now with download speed!)
2021-05-18 22:45:19 +02:00
595de88d96
Fix authenticator and crawler names
...
Now, the "auth:" and "crawl:" parts are considered part of the name. This fixes
crawlers not being able to find their authenticators.
2021-05-15 15:25:05 +02:00
0acdee15a0
Let crawlers obtain authenticators
2021-05-13 18:57:20 +02:00
0459ed093e
Add simple authenticator
...
... including some required authenticator infrastructure
2021-05-11 00:28:03 +02:00
d5f29f01c5
Use global conductor instance
...
The switch from crawler-local conductors to a single pferd-global conductor was
made to prepare for auth section credential providers.
2021-05-11 00:05:04 +02:00
60cd9873bc
Add local file crawler
2021-05-06 01:02:40 +02:00
bbfdadc463
Implement output directory
2021-05-05 18:08:34 +02:00
91c33596da
Load crawlers from config file
2021-04-30 16:22:14 +02:00
3ea86d18a0
Jerry-rig DummyCrawler to run
2021-04-29 13:45:04 +02:00
f47b137b59
Fix ILIAS init.py and Pferd.py authenticators
2020-12-06 13:15:32 +01:00
9f6dc56a7b
Use a strategy to decide conflict resolution
2020-12-02 19:32:57 +01:00
f3a4663491
Add passive/no_prompt flag
2020-12-02 18:24:07 +01:00
ef343dec7c
Merge organizer download summaries
2020-11-04 15:06:58 +01:00
f4abe3197c
Add ipd crawler
2020-11-03 21:15:40 +01:00
51a713fa04
Allow crawling courses or folders with sync_url
...
Video folders do not work, if they are passed directly. Their containing
folder must be specified instead.
2020-09-28 20:00:01 +02:00
a57ee8b96b
Add timeout to video downloads to work around requests IPv6 bug
2020-08-11 14:40:30 +02:00
77874b432b
Also add personal_desktop to download summary
2020-07-15 22:47:44 +02:00
d9ea688145
Use pretty logger for summaries
2020-06-26 17:24:36 +02:00
402ae81335
Fix type hints
2020-06-26 13:17:44 +00:00
739522a151
Move download summary into a separate class
2020-06-25 23:07:11 +02:00
6c034209b6
Add deleted files to summary
2020-06-25 22:00:28 +02:00
f6fbd5e4bb
Add download summary
2020-06-25 19:19:34 +02:00
b969a1854a
Remove unneeded whitespace
2020-05-30 00:22:31 +02:00
072c6630bf
Avoid logging import in config
2020-05-12 18:19:23 +00:00
4f56c8f192
Pass element type to ilias directory filter
2020-05-12 14:41:13 +02:00
9850ab1d73
Allow crawling the ILIAS Personal Desktop
2020-05-10 12:16:42 +02:00
9950144e97
Allow passing a playlist URL to diva instead of an id
2020-05-10 11:17:13 +02:00
f6faacabb0
Move FatalException to errors.py
2020-05-09 00:11:21 +02:00
19c1e3ac6f
Fail on invalid ILIAS course ids
2020-05-09 00:11:20 +02:00
afa48c2d2d
Swallow and print errors instead of crashing
2020-05-09 00:10:54 +02:00
bee3d70998
Added a diva playlist downloader
2020-04-30 17:18:45 +02:00
8a42a2a396
Move logging into its own file
2020-04-25 20:02:01 +02:00
1aaa6e7ab5
Use PathLike everywhere
2020-04-24 18:41:14 +00:00
7f53543324
Satisfy pylint and add todo
2020-04-24 18:26:28 +00:00
8258fa8919
Add test run option to PFERD
2020-04-24 18:00:21 +00:00
5b929f09a2
Move download strategies to downloader
...
Also fixes an issue where the downloader didn't mark files that were not
downloaded due to the strategy used.
2020-04-24 14:27:40 +00:00
4d32f863bc
Clean up organizer after synchronizing
2020-04-24 14:17:23 +00:00
4e7333b396
Allow specifying paths as strings in Pferd
2020-04-24 11:50:40 +00:00
4c0e3b493a
Use download_modified_or_new as default strategy
2020-04-24 13:48:06 +02:00
509e624d47
Satisfy pyling. Useful docstrings? Not quite sure.
2020-04-23 20:35:59 +02:00
0b00a9c26b
Log when starting to synchronize
2020-04-23 19:56:37 +02:00
5ef5a56e69
Extract Location into separate file
2020-04-23 17:38:28 +00:00
f3f4be2690
More free functions
2020-04-23 19:21:49 +02:00
076b8c5a1f
Add download strategies to save bandwith
...
Only download files that are newer than the local version.
2020-04-23 18:29:20 +02:00
