I-Al-Istannen
b97b6fae6b
Update minimum Python version to 3.11
2025-04-15 21:35:20 +02:00
I-Al-Istannen
be175f9347
Download only new/updated forum threads
2025-02-19 16:16:37 +01:00
Tim
26e802d88b
Add clickable links to file names in the printed report ( #100 )
...
Co-authored-by: I-Al-Istannen <i-al-istannen@users.noreply.github.com >
2024-11-04 00:32:32 +01:00
Tim
739dd95850
Use Last-Modified and ETag headers to determine KIT-IPD file versions ( #95 )
...
Co-authored-by: I-Al-Istannen <i-al-istannen@users.noreply.github.com >
2024-10-27 19:03:47 +01:00
Joscha
602044ff1b
Fix mypy errors and add missing await
2022-04-27 22:52:50 +02:00
I-Al-Istannen
a82a0b19c2
Collect crawler warnings/errors and include them in the report
2021-11-07 21:48:55 +01:00
Joscha
544d45cbc5
Catch non-critical exceptions at crawler top level
2021-07-13 15:42:11 +02:00
I-Al-Istannen
91200f3684
Fix nondeterministic name deduplication
2021-07-03 12:09:55 +02:00
Joscha
df3ad3d890
Add 'skip' option to crawlers
2021-06-04 18:47:13 +02:00
Joscha
7b062883f6
Use raw paths for --debug-transforms
...
Previously, the already-transformed paths were used, which meant that
--debug-transforms was cumbersome to use (as you had to remove all transforms
and crawl once before getting useful results).
2021-05-31 12:33:37 +02:00
Joscha
64a2960751
Align paths in status messages and progress bars
...
Also print "Ignored" when paths are ignored due to transforms
2021-05-31 12:32:42 +02:00
Joscha
474aa7e1cc
Use sorted path order when debugging transforms
2021-05-27 15:41:00 +00:00
Joscha
533f75ea71
Add --debug-transforms flag
2021-05-26 11:37:32 +02:00
Joscha
61430c8739
Overhaul config and CLI option names
2021-05-25 14:23:38 +02:00
Joscha
eb8b915813
Fix path prefix on windows
...
Previously, the path prefix was only set if "windows_paths" was true, regardless
of OS. Now the path prefix is always set on windows and never set on other OSes.
2021-05-25 14:23:38 +02:00
Joscha
bce3dc384d
Deduplicate path names in crawler
...
Also rename files so they follow the restrictions for windows file names if
we're on windows.
2021-05-25 12:11:15 +02:00
Joscha
27b5a8e490
Rename log.action to log.status
2021-05-23 22:40:33 +02:00
Joscha
ce1dbda5b4
Overhaul colours
...
"Crawled" and "Downloaded" are now printed less bright than "Crawling" and
"Downloading" as they're not as important. Explain topics are printed in yellow
to stand out a bit more from the cyan action messages.
2021-05-23 21:33:04 +02:00
Joscha
6ca0ecdf05
Load and store reports
2021-05-23 20:46:29 +02:00
Joscha
2fdf24495b
Restructure crawling and auth related modules
2021-05-23 19:16:42 +02:00
