9 Commits

Author	SHA1	Message	Date
Joscha	b0f731bf84	Make crawlers use transformers	2021-05-15 15:25:05 +02:00
Joscha	302b8c0c34	Fix errors loading local crawler config ... Apparently getint and getfloat may return a None even though this is not mentioned in their type annotations.	2021-05-15 15:25:05 +02:00
Joscha	ed2e19a150	Add reasons for invalid values	2021-05-15 15:25:05 +02:00
Joscha	1591cb9197	Add options to slow down local crawler ... These options are meant to make the local crawler behave more like a network-based crawler for purposes of testing and debugging other parts of the code base.	2021-05-15 15:25:01 +02:00
Joscha	94d6a01cca	Use file mtime in local crawler	2021-05-13 19:42:40 +02:00
Joscha	c3ce6bb31c	Fix crawler cleanup not being awaited	2021-05-11 00:28:45 +02:00
Joscha	d5f29f01c5	Use global conductor instance ... The switch from crawler-local conductors to a single pferd-global conductor was made to prepare for auth section credential providers.	2021-05-11 00:05:04 +02:00
Joscha	f9b2fd60e2	Document local crawler and auth	2021-05-09 01:33:47 +02:00
Joscha	60cd9873bc	Add local file crawler	2021-05-06 01:02:40 +02:00