mirrors/pferd
1
0
Fork 0
mirror of https://github.com/Garmelon/PFERD.git synced 2023-12-21 10:23:01 +01:00
Code Issues Packages Projects Releases Wiki Activity

Deduplicate path names in crawler

Browse Source 
Also rename files so they follow the restrictions for windows file names if
we're on windows.
This commit is contained in:
Joscha
2021-05-25 11:58:01 +02:00
parent c21ddf225b
commit bce3dc384d
6 changed files with 114 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
CONFIG.md
View File

@ -83,6 +83,8 @@ crawlers:
- `delay_between_tasks`: Time (in seconds) that the crawler should wait between
subsequent tasks. Can be used as a sort of rate limit to avoid unnecessary
load for the crawl target. (Default: 0.0)
- `windows_paths`: Whether PFERD should find alternative names for paths that
are invalid on Windows. (Default: `yes` on Windows, `no` otherwise)
Some crawlers may also require credentials for authentication. To configure how
the crawler obtains its credentials, the `auth` option is used. It is set to the