mirrors/pferd
1
0
Fork 0
mirror of https://github.com/Garmelon/PFERD.git synced 2023-12-21 10:23:01 +01:00
Code Issues Packages Projects Releases Wiki Activity

Add options to slow down local crawler

Browse Source 
These options are meant to make the local crawler behave more like a
network-based crawler for purposes of testing and debugging other parts of the
code base.
This commit is contained in:
Joscha
2021-05-14 21:41:24 +02:00
parent 0c9167512c
commit 1591cb9197
3 changed files with 65 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
CONFIG.md
View File

@ -102,9 +102,15 @@ authenticators is `type`:
### The `local` crawler
This crawler crawls a local directory. It is really simple and mostly useful for
testing different setups.
testing different setups. The various delay options are meant to make the
crawler simulate a slower, network-based crawler.
- `path`: Path to the local directory to crawl. (Required)
- `crawl_delay`: Maximum artificial delay (in seconds) to simulate for crawl
requests. (Optional)
- `download_delay`: Maximum artificial delay (in seconds) to simulate for
download requests. (Optional)
- `download_speed`: Download speed (in bytes per second) to simulate. (Optional)
## Authenticator types
@ -114,8 +120,8 @@ With this authenticator, the username and password can be set directly in the
config file. If the username or password are not specified, the user is prompted
via the terminal.
- `username`: The username (Optional)
- `password`: The password (Optional)
- `username`: The username. (Optional)
- `password`: The password. (Optional)
## Transformation rules