9 Commits

Author	SHA1	Message	Date
Joscha	acd674f0a0	Change limiter logic ... Now download tasks are a subset of all tasks.	2021-05-15 15:25:05 +02:00
Joscha	296a169dd3	Make limiter logic more complex ... The limiter can now distinguish between crawl and download actions and has a fancy slot system and delay logic.	2021-05-15 15:25:05 +02:00
Joscha	1591cb9197	Add options to slow down local crawler ... These options are meant to make the local crawler behave more like a network-based crawler for purposes of testing and debugging other parts of the code base.	2021-05-15 15:25:01 +02:00
Joscha	961f40f9a1	Document simple authenticator	2021-05-13 19:55:04 +02:00
Joscha	f9b2fd60e2	Document local crawler and auth	2021-05-09 01:33:47 +02:00
Joscha	fde811ae5a	Document on_conflict option	2021-05-05 12:24:35 +02:00
Joscha	a8dcf941b9	Document possible redownload settings	2021-04-30 15:32:56 +02:00
Joscha	e7a51decb0	Elaborate on transforms and implement changes	2021-04-29 20:24:18 +02:00
Joscha	9ec19be113	Document config file format	2021-04-29 20:24:18 +02:00