13 Commits

Author	SHA1	Message	Date
Joscha	1591cb9197	Add options to slow down local crawler ... These options are meant to make the local crawler behave more like a network-based crawler for purposes of testing and debugging other parts of the code base.	2021-05-15 15:25:01 +02:00
Joscha	94d6a01cca	Use file mtime in local crawler	2021-05-13 19:42:40 +02:00
Joscha	0acdee15a0	Let crawlers obtain authenticators	2021-05-13 18:57:20 +02:00
Joscha	c3ce6bb31c	Fix crawler cleanup not being awaited	2021-05-11 00:28:45 +02:00
Joscha	d5f29f01c5	Use global conductor instance ... The switch from crawler-local conductors to a single pferd-global conductor was made to prepare for auth section credential providers.	2021-05-11 00:05:04 +02:00
Joscha	595ba8b7ab	Remove dummy crawler	2021-05-10 23:47:46 +02:00
Joscha	f9b2fd60e2	Document local crawler and auth	2021-05-09 01:33:47 +02:00
Joscha	60cd9873bc	Add local file crawler	2021-05-06 01:02:40 +02:00
Joscha	273d56c39a	Properly load crawler config	2021-05-05 23:45:10 +02:00
Joscha	f776186480	Use PurePath instead of Path ... Path should only be used when we need to access the file system. For all other purposes (mainly crawling), we use PurePath instead since the paths don't correspond to paths in the local file system.	2021-04-29 20:20:25 +02:00
Joscha	d96a361325	Test and fix exclusive output	2021-04-29 15:27:16 +02:00
Joscha	ac3bfd7388	Make progress bars easier to use ... The crawler now supports two types of progress bars	2021-04-29 13:53:16 +02:00
Joscha	bbc792f9fb	Implement Crawler and DummyCrawler	2021-04-29 13:44:29 +02:00