mirrors/pferd
1
0
Fork 0
mirror of https://github.com/Garmelon/PFERD.git synced 2023-12-21 10:23:01 +01:00
Code Issues Packages Projects Releases Wiki Activity

Add kit-ipd crawler

Browse Source
This commit is contained in:
I-Al-Istannen
2021-10-21 12:01:41 +02:00
parent 742632ed8d
commit 6673077397
6 changed files with 196 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
CONFIG.md
View File

@ -136,6 +136,13 @@ crawler simulate a slower, network-based crawler.
requests. (Default: `0.0`)
- `download_speed`: Download speed (in bytes per second) to simulate. (Optional)
### The `kit-ipd` crawler
This crawler crals a KIT ipd page by url. The root page can be crawled from
outside the KIT network so you will be informed about any new/deleted files,
but downloading files requires you to be within. Adding a show delay between
requests is likely a good idea.
### The `kit-ilias-web` crawler
This crawler crawls the KIT ILIAS instance.