4 Commits

Author	SHA1	Message	Date
I-Al-Istannen	83d12fcf2d	Add some explains to ilias crawler and use crawler exceptions	2021-05-20 14:58:54 +02:00
I-Al-Istannen	e4f9560655	Only retry on aiohttp errors in ILIAS crawler ... This patch removes quite a few retries and now only retries the ilias element method. Every other HTTP-interacting method (except for the root requests) is called from there and should be covered. In the future we also want to retry the root a few times, but that will be done after the download sink API is adjusted.	2021-05-19 22:01:09 +02:00
I-Al-Istannen	8cfa818f04	Only call should_crawl once	2021-05-19 21:57:55 +02:00
I-Al-Istannen	81301f3a76	Rename the ilias crawler to ilias web crawler	2021-05-19 21:41:17 +02:00