29251fa003
Merge 77c1f1516c into e41a22149e
2023-08-26 17:16:00 +02:00
e41a22149e
Add default show-not-deleted option
...
If set to `no`, PFERD won't print status or report messages for not deleted files
2023-08-26 17:13:45 +02:00
a82a0b19c2
Collect crawler warnings/errors and include them in the report
2021-11-07 21:48:55 +01:00
77c1f1516c
Used proper plural
2021-11-02 12:41:40 +01:00
9e12e96d90
Added alias functionality
2021-11-02 03:42:08 +01:00
80eeb8fe97
Add --skip option
2021-07-01 11:02:21 +02:00
df3ad3d890
Add 'skip' option to crawlers
2021-06-04 18:47:13 +02:00
533f75ea71
Add --debug-transforms flag
2021-05-26 11:37:32 +02:00
adb5d4ade3
Print files that are *not* deleted by cleanup
...
These are files that are not present on the remote source any more, but still
present locally. They also show up in the report.
2021-05-26 10:58:19 +02:00
5a331663e4
Rename functions for consistency
2021-05-25 15:49:06 +02:00
f68849c65f
Fix rules not being parsed entirely
2021-05-25 15:42:46 +02:00
edb52a989e
Print report even if exiting due to Ctrl+C
2021-05-25 15:35:36 +02:00
22c2259adb
Clean up authenticator exceptions
...
- Renamed to *Error for consistency
- Treating AuthError like CrawlError
2021-05-25 14:23:38 +02:00
1c1f781be4
Reword some log messages
2021-05-24 13:17:28 +02:00
c687d4a51a
Implement cookie sharing
2021-05-24 13:10:44 +02:00
9cce78669f
Print report after all crawlers have finished
2021-05-23 21:17:13 +02:00
2fdf24495b
Restructure crawling and auth related modules
2021-05-23 19:16:42 +02:00
a9af56a5e9
Improve specifying crawlers via CLI
...
Instead of removing the sections of unselected crawlers from the config file,
crawler selection now happens in the Pferd after loading the crawlers and is
more sophisticated. It also has better error messages.
2021-05-23 18:18:50 +02:00
662191eca9
Fix crash as soon as first cl or dl token was acquired
2021-05-22 20:25:58 +00:00
9889ce6b57
Improve PFERD error handling
2021-05-22 21:13:25 +02:00
4b68fa771f
Move logging logic to singleton
...
- Renamed module and class because "conductor" didn't make a lot of sense
- Used singleton approach (there's only one stdout after all)
- Redesigned progress bars (now with download speed!)
2021-05-18 22:45:19 +02:00
595de88d96
Fix authenticator and crawler names
...
Now, the "auth:" and "crawl:" parts are considered part of the name. This fixes
crawlers not being able to find their authenticators.
2021-05-15 15:25:05 +02:00
0acdee15a0
Let crawlers obtain authenticators
2021-05-13 18:57:20 +02:00
0459ed093e
Add simple authenticator
...
... including some required authenticator infrastructure
2021-05-11 00:28:03 +02:00
d5f29f01c5
Use global conductor instance
...
The switch from crawler-local conductors to a single pferd-global conductor was
made to prepare for auth section credential providers.
2021-05-11 00:05:04 +02:00
60cd9873bc
Add local file crawler
2021-05-06 01:02:40 +02:00
bbfdadc463
Implement output directory
2021-05-05 18:08:34 +02:00
91c33596da
Load crawlers from config file
2021-04-30 16:22:14 +02:00
3ea86d18a0
Jerry-rig DummyCrawler to run
2021-04-29 13:45:04 +02:00
f47b137b59
Fix ILIAS init.py and Pferd.py authenticators
2020-12-06 13:15:32 +01:00
9f6dc56a7b
Use a strategy to decide conflict resolution
2020-12-02 19:32:57 +01:00
f3a4663491
Add passive/no_prompt flag
2020-12-02 18:24:07 +01:00
ef343dec7c
Merge organizer download summaries
2020-11-04 15:06:58 +01:00
f4abe3197c
Add ipd crawler
2020-11-03 21:15:40 +01:00
51a713fa04
Allow crawling courses or folders with sync_url
...
Video folders do not work, if they are passed directly. Their containing
folder must be specified instead.
2020-09-28 20:00:01 +02:00
a57ee8b96b
Add timeout to video downloads to work around requests IPv6 bug
2020-08-11 14:40:30 +02:00
77874b432b
Also add personal_desktop to download summary
2020-07-15 22:47:44 +02:00
d9ea688145
Use pretty logger for summaries
2020-06-26 17:24:36 +02:00
402ae81335
Fix type hints
2020-06-26 13:17:44 +00:00
739522a151
Move download summary into a separate class
2020-06-25 23:07:11 +02:00
6c034209b6
Add deleted files to summary
2020-06-25 22:00:28 +02:00
f6fbd5e4bb
Add download summary
2020-06-25 19:19:34 +02:00
b969a1854a
Remove unneeded whitespace
2020-05-30 00:22:31 +02:00
072c6630bf
Avoid logging import in config
2020-05-12 18:19:23 +00:00
4f56c8f192
Pass element type to ilias directory filter
2020-05-12 14:41:13 +02:00
9850ab1d73
Allow crawling the ILIAS Personal Desktop
2020-05-10 12:16:42 +02:00
9950144e97
Allow passing a playlist URL to diva instead of an id
2020-05-10 11:17:13 +02:00
f6faacabb0
Move FatalException to errors.py
2020-05-09 00:11:21 +02:00
19c1e3ac6f
Fail on invalid ILIAS course ids
2020-05-09 00:11:20 +02:00
afa48c2d2d
Swallow and print errors instead of crashing
2020-05-09 00:10:54 +02:00
