mirrors/pferd
1
0
Fork 0
mirror of https://github.com/Garmelon/PFERD.git synced 2023-12-21 10:23:01 +01:00
Code Issues Packages Projects Releases Wiki Activity
471 Commits 6 Branches 43 Tags 1.6 MiB
c687d4a51a
Commit Graph

53 Commits

Author SHA1 Message Date
Joscha
c687d4a51a Implement cookie sharing 2021-05-24 13:10:44 +02:00
Joscha
9cce78669f Print report after all crawlers have finished 2021-05-23 21:17:13 +02:00
Joscha
2fdf24495b Restructure crawling and auth related modules 2021-05-23 19:16:42 +02:00
Joscha
a9af56a5e9 Improve specifying crawlers via CLI 
Instead of removing the sections of unselected crawlers from the config file,
crawler selection now happens in the Pferd after loading the crawlers and is
more sophisticated. It also has better error messages.
 2021-05-23 18:18:50 +02:00
Joscha
662191eca9 Fix crash as soon as first cl or dl token was acquired 2021-05-22 20:25:58 +00:00
Joscha
9889ce6b57 Improve PFERD error handling 2021-05-22 21:13:25 +02:00
Joscha
4b68fa771f Move logging logic to singleton 
- Renamed module and class because "conductor" didn't make a lot of sense
- Used singleton approach (there's only one stdout after all)
- Redesigned progress bars (now with download speed!)
 2021-05-18 22:45:19 +02:00
Joscha
595de88d96 Fix authenticator and crawler names 
Now, the "auth:" and "crawl:" parts are considered part of the name. This fixes
crawlers not being able to find their authenticators.
 2021-05-15 15:25:05 +02:00
Joscha
0acdee15a0 Let crawlers obtain authenticators 2021-05-13 18:57:20 +02:00
Joscha
0459ed093e Add simple authenticator 
... including some required authenticator infrastructure
 2021-05-11 00:28:03 +02:00
Joscha
d5f29f01c5 Use global conductor instance 
The switch from crawler-local conductors to a single pferd-global conductor was
made to prepare for auth section credential providers.
 2021-05-11 00:05:04 +02:00
Joscha
60cd9873bc Add local file crawler 2021-05-06 01:02:40 +02:00
Joscha
bbfdadc463 Implement output directory 2021-05-05 18:08:34 +02:00
Joscha
91c33596da Load crawlers from config file 2021-04-30 16:22:14 +02:00
Joscha
3ea86d18a0 Jerry-rig DummyCrawler to run 2021-04-29 13:45:04 +02:00
I-Al-Istannen
f47b137b59 Fix ILIAS init.py and Pferd.py authenticators 2020-12-06 13:15:32 +01:00
I-Al-Istannen
9f6dc56a7b Use a strategy to decide conflict resolution 2020-12-02 19:32:57 +01:00
Christophe
f3a4663491 Add passive/no_prompt flag 2020-12-02 18:24:07 +01:00
I-Al-Istannen
ef343dec7c Merge organizer download summaries 2020-11-04 15:06:58 +01:00
I-Al-Istannen
f4abe3197c Add ipd crawler 2020-11-03 21:15:40 +01:00
I-Al-Istannen
51a713fa04 Allow crawling courses or folders with sync_url 
Video folders do not work, if they are passed directly. Their containing
folder must be specified instead.
 2020-09-28 20:00:01 +02:00
I-Al-Istannen
a57ee8b96b Add timeout to video downloads to work around requests IPv6 bug 2020-08-11 14:40:30 +02:00
I-Al-Istannen
77874b432b Also add personal_desktop to download summary 2020-07-15 22:47:44 +02:00
I-Al-Istannen
d9ea688145 Use pretty logger for summaries 2020-06-26 17:24:36 +02:00
Joscha
402ae81335 Fix type hints 2020-06-26 13:17:44 +00:00
Daniel Augustin
739522a151
Move download summary into a separate class 2020-06-25 23:07:11 +02:00
Daniel Augustin
6c034209b6
Add deleted files to summary 2020-06-25 22:00:28 +02:00
Daniel Augustin
f6fbd5e4bb
Add download summary 2020-06-25 19:19:34 +02:00
I-Al-Istannen
b969a1854a Remove unneeded whitespace 2020-05-30 00:22:31 +02:00
Joscha
072c6630bf Avoid logging import in config 2020-05-12 18:19:23 +00:00
I-Al-Istannen
4f56c8f192 Pass element type to ilias directory filter 2020-05-12 14:41:13 +02:00
I-Al-Istannen
9850ab1d73 Allow crawling the ILIAS Personal Desktop 2020-05-10 12:16:42 +02:00
I-Al-Istannen
9950144e97 Allow passing a playlist URL to diva instead of an id 2020-05-10 11:17:13 +02:00
I-Al-Istannen
f6faacabb0 Move FatalException to errors.py 2020-05-09 00:11:21 +02:00
I-Al-Istannen
19c1e3ac6f Fail on invalid ILIAS course ids 2020-05-09 00:11:20 +02:00
I-Al-Istannen
afa48c2d2d Swallow and print errors instead of crashing 2020-05-09 00:10:54 +02:00
I-Al-Istannen
bee3d70998 Added a diva playlist downloader 2020-04-30 17:18:45 +02:00
I-Al-Istannen
8a42a2a396 Move logging into its own file 2020-04-25 20:02:01 +02:00
Joscha
1aaa6e7ab5 Use PathLike everywhere 2020-04-24 18:41:14 +00:00
Joscha
7f53543324 Satisfy pylint and add todo 2020-04-24 18:26:28 +00:00
Joscha
8258fa8919 Add test run option to PFERD 2020-04-24 18:00:21 +00:00
Joscha
5b929f09a2 Move download strategies to downloader 
Also fixes an issue where the downloader didn't mark files that were not
downloaded due to the strategy used.
 2020-04-24 14:27:40 +00:00
Joscha
4d32f863bc Clean up organizer after synchronizing 2020-04-24 14:17:23 +00:00
Joscha
4e7333b396 Allow specifying paths as strings in Pferd 2020-04-24 11:50:40 +00:00
I-Al-Istannen
4c0e3b493a Use download_modified_or_new as default strategy 2020-04-24 13:48:06 +02:00
I-Al-Istannen
509e624d47 Satisfy pyling. Useful docstrings? Not quite sure. 2020-04-23 20:35:59 +02:00
I-Al-Istannen
0b00a9c26b Log when starting to synchronize 2020-04-23 19:56:37 +02:00
Joscha
5ef5a56e69 Extract Location into separate file 2020-04-23 17:38:28 +00:00
I-Al-Istannen
f3f4be2690 More free functions 2020-04-23 19:21:49 +02:00
I-Al-Istannen
076b8c5a1f Add download strategies to save bandwith 
Only download files that are newer than the local version.
 2020-04-23 18:29:20 +02:00