6 Commits

Author	SHA1	Message	Date
I-Al-Istannen	83d12fcf2d	Add some explains to ilias crawler and use crawler exceptions	2021-05-20 14:58:54 +02:00
I-Al-Istannen	e4f9560655	Only retry on aiohttp errors in ILIAS crawler ... This patch removes quite a few retries and now only retries the ilias element method. Every other HTTP-interacting method (except for the root requests) is called from there and should be covered. In the future we also want to retry the root a few times, but that will be done after the download sink API is adjusted.	2021-05-19 22:01:09 +02:00
I-Al-Istannen	8cfa818f04	Only call should_crawl once	2021-05-19 21:57:55 +02:00
I-Al-Istannen	81301f3a76	Rename the ilias crawler to ilias web crawler	2021-05-19 21:41:17 +02:00
I-Al-Istannen	2976b4d352	Move ILIAS file templates to own file	2021-05-19 21:37:10 +02:00
I-Al-Istannen	9f03702e69	Split up ilias crawler in multiple files ... The ilias crawler contained a crawler and an HTML parser, now they are split in two.	2021-05-19 21:34:36 +02:00