mirrors/pferd
1
0
Fork 0
mirror of https://github.com/Garmelon/PFERD.git synced 2025-07-20 01:42:37 +02:00
Code Issues Packages Projects Releases Wiki Activity
399 Commits 9 Branches 51 Tags
e4f9560655b2bd8f56a77f0b126d14b1db61b52c
Commit Graph

5 Commits

Author SHA1 Message Date
I-Al-Istannen
e4f9560655 Only retry on aiohttp errors in ILIAS crawler 
This patch removes quite a few retries and now only retries the ilias
element method. Every other HTTP-interacting method (except for the root
requests) is called from there and should be covered.

In the future we also want to retry the root a few times, but that
will be done after the download sink API is adjusted.
 2021-05-19 22:01:09 +02:00
I-Al-Istannen
8cfa818f04 Only call should_crawl once 2021-05-19 21:57:55 +02:00
I-Al-Istannen
81301f3a76 Rename the ilias crawler to ilias web crawler 2021-05-19 21:41:17 +02:00
I-Al-Istannen
2976b4d352 Move ILIAS file templates to own file 2021-05-19 21:37:10 +02:00
I-Al-Istannen
9f03702e69 Split up ilias crawler in multiple files 
The ilias crawler contained a crawler and an HTML parser, now they are
split in two.
 2021-05-19 21:34:36 +02:00