20 Commits

Author	SHA1	Message	Date
Joscha	296a169dd3	Make limiter logic more complex ... The limiter can now distinguish between crawl and download actions and has a fancy slot system and delay logic.	2021-05-15 15:25:05 +02:00
Joscha	6e5fdf4e9e	Set user agent to "pferd/<version>"	2021-05-14 21:27:44 +02:00
Joscha	d565df27b3	Add HttpCrawler	2021-05-13 22:28:14 +02:00
Joscha	68781a88ab	Fix asynchronous methods being not awaited	2021-05-13 19:39:49 +02:00
Joscha	0acdee15a0	Let crawlers obtain authenticators	2021-05-13 18:57:20 +02:00
Joscha	d5f29f01c5	Use global conductor instance ... The switch from crawler-local conductors to a single pferd-global conductor was made to prepare for auth section credential providers.	2021-05-11 00:05:04 +02:00
Joscha	cec0a8e1fc	Fix mymy errors	2021-05-09 01:45:01 +02:00
Joscha	60cd9873bc	Add local file crawler	2021-05-06 01:02:40 +02:00
Joscha	273d56c39a	Properly load crawler config	2021-05-05 23:45:10 +02:00
Joscha	5497dd2827	Add @noncritical and @repeat decorators	2021-05-05 23:36:54 +02:00
Joscha	bbfdadc463	Implement output directory	2021-05-05 18:08:34 +02:00
Joscha	91c33596da	Load crawlers from config file	2021-04-30 16:22:14 +02:00
Joscha	f776186480	Use PurePath instead of Path ... Path should only be used when we need to access the file system. For all other purposes (mainly crawling), we use PurePath instead since the paths don't correspond to paths in the local file system.	2021-04-29 20:20:25 +02:00
Joscha	502654d853	Fix mypy errors	2021-04-29 15:47:52 +02:00
Joscha	d2103d7c44	Document crawler	2021-04-29 15:43:20 +02:00
Joscha	d96a361325	Test and fix exclusive output	2021-04-29 15:27:16 +02:00
Joscha	2e85d26b6b	Use conductor via context manager	2021-04-29 14:23:28 +02:00
Joscha	6431a3fb3d	Fix some mypy errors	2021-04-29 14:23:09 +02:00
Joscha	ac3bfd7388	Make progress bars easier to use ... The crawler now supports two types of progress bars	2021-04-29 13:53:16 +02:00
Joscha	bbc792f9fb	Implement Crawler and DummyCrawler	2021-04-29 13:44:29 +02:00