Mr. Pine
dbc2553b11
Add default show-not-deleted option
...
If set to `no`, PFERD won't print status or report messages for not deleted files
2023-08-26 18:43:01 +02:00
I-Al-Istannen
a709280cbf
Try to detect unsupported config file encoding
...
The encoding detection is quite rudimentary, but should detect the
default windows encoding in many cases.
2022-04-27 22:03:47 +02:00
I-Al-Istannen
a99ddaa0cc
Read and write config in UTF-8
2022-04-27 21:47:51 +02:00
Joscha
e1bda94329
Load credential file from correct path
2021-06-01 09:18:08 +00:00
Joscha
5a331663e4
Rename functions for consistency
2021-05-25 15:49:06 +02:00
Joscha
edb52a989e
Print report even if exiting due to Ctrl+C
2021-05-25 15:35:36 +02:00
Joscha
1c1f781be4
Reword some log messages
2021-05-24 13:17:28 +02:00
Joscha
c687d4a51a
Implement cookie sharing
2021-05-24 13:10:44 +02:00
Joscha
edbd92dbbf
Add --status and --report flags
2021-05-23 22:41:59 +02:00
Joscha
803e5628a2
Clean up logging
...
Paths are now (hopefully) logged consistently across all crawlers
2021-05-23 11:37:19 +02:00
Joscha
c88f20859a
Explain config file dumping
2021-05-23 11:04:50 +02:00
Joscha
54dd2f8337
Clean up main and improve error handling
2021-05-22 16:47:24 +02:00
Joscha
3300886120
Explain config file loading
2021-05-19 18:11:43 +02:00
Joscha
0d10752b5a
Configure explain log level via cli and config file
2021-05-19 17:50:10 +02:00
Joscha
05573ccc53
Add fancy CLI options
2021-05-15 22:22:01 +02:00
Joscha
b70b62cef5
Make crawler sections start with "crawl:"
...
Also, use only the part of the section name after the "crawl:" as the crawler's
output directory. Now, the implementation matches the documentation again
2021-05-15 17:24:37 +02:00
Joscha
595de88d96
Fix authenticator and crawler names
...
Now, the "auth:" and "crawl:" parts are considered part of the name. This fixes
crawlers not being able to find their authenticators.
2021-05-15 15:25:05 +02:00
Joscha
296a169dd3
Make limiter logic more complex
...
The limiter can now distinguish between crawl and download actions and has a
fancy slot system and delay logic.
2021-05-15 15:25:05 +02:00
Joscha
0459ed093e
Add simple authenticator
...
... including some required authenticator infrastructure
2021-05-11 00:28:03 +02:00
Joscha
d5f29f01c5
Use global conductor instance
...
The switch from crawler-local conductors to a single pferd-global conductor was
made to prepare for auth section credential providers.
2021-05-11 00:05:04 +02:00
Joscha
273d56c39a
Properly load crawler config
2021-05-05 23:45:10 +02:00
Joscha
91c33596da
Load crawlers from config file
2021-04-30 16:22:14 +02:00
Joscha
fbebc46c58
Load and dump config
2021-04-29 09:51:50 +02:00
