5 Commits

Author	SHA1	Message	Date
I-Al-Istannen	4b104b6252	Try out some HTTP authentication handling ... This is by no means final yet and will change a bit once the dl and cl are changed, but it might serve as a first try. It is also wholly untested.	2021-05-21 12:02:51 +02:00
I-Al-Istannen	83d12fcf2d	Add some explains to ilias crawler and use crawler exceptions	2021-05-20 14:58:54 +02:00
I-Al-Istannen	e4f9560655	Only retry on aiohttp errors in ILIAS crawler ... This patch removes quite a few retries and now only retries the ilias element method. Every other HTTP-interacting method (except for the root requests) is called from there and should be covered. In the future we also want to retry the root a few times, but that will be done after the download sink API is adjusted.	2021-05-19 22:01:09 +02:00
I-Al-Istannen	8cfa818f04	Only call should_crawl once	2021-05-19 21:57:55 +02:00
I-Al-Istannen	81301f3a76	Rename the ilias crawler to ilias web crawler	2021-05-19 21:41:17 +02:00