b97b6fae6b
Update minimum Python version to 3.11
2025-04-15 21:35:20 +02:00
be175f9347
Download only new/updated forum threads
2025-02-19 16:16:37 +01:00
26e802d88b
Add clickable links to file names in the printed report ( #100 )
...
Co-authored-by: I-Al-Istannen <i-al-istannen@users.noreply.github.com >
2024-11-04 00:32:32 +01:00
739dd95850
Use Last-Modified and ETag headers to determine KIT-IPD file versions ( #95 )
...
Co-authored-by: I-Al-Istannen <i-al-istannen@users.noreply.github.com >
2024-10-27 19:03:47 +01:00
602044ff1b
Fix mypy errors and add missing await
2022-04-27 22:52:50 +02:00
a82a0b19c2
Collect crawler warnings/errors and include them in the report
2021-11-07 21:48:55 +01:00
544d45cbc5
Catch non-critical exceptions at crawler top level
2021-07-13 15:42:11 +02:00
91200f3684
Fix nondeterministic name deduplication
2021-07-03 12:09:55 +02:00
df3ad3d890
Add 'skip' option to crawlers
2021-06-04 18:47:13 +02:00
7b062883f6
Use raw paths for --debug-transforms
...
Previously, the already-transformed paths were used, which meant that
--debug-transforms was cumbersome to use (as you had to remove all transforms
and crawl once before getting useful results).
2021-05-31 12:33:37 +02:00
64a2960751
Align paths in status messages and progress bars
...
Also print "Ignored" when paths are ignored due to transforms
2021-05-31 12:32:42 +02:00
474aa7e1cc
Use sorted path order when debugging transforms
2021-05-27 15:41:00 +00:00
533f75ea71
Add --debug-transforms flag
2021-05-26 11:37:32 +02:00
61430c8739
Overhaul config and CLI option names
2021-05-25 14:23:38 +02:00
eb8b915813
Fix path prefix on windows
...
Previously, the path prefix was only set if "windows_paths" was true, regardless
of OS. Now the path prefix is always set on windows and never set on other OSes.
2021-05-25 14:23:38 +02:00
bce3dc384d
Deduplicate path names in crawler
...
Also rename files so they follow the restrictions for windows file names if
we're on windows.
2021-05-25 12:11:15 +02:00
27b5a8e490
Rename log.action to log.status
2021-05-23 22:40:33 +02:00
ce1dbda5b4
Overhaul colours
...
"Crawled" and "Downloaded" are now printed less bright than "Crawling" and
"Downloading" as they're not as important. Explain topics are printed in yellow
to stand out a bit more from the cyan action messages.
2021-05-23 21:33:04 +02:00
6ca0ecdf05
Load and store reports
2021-05-23 20:46:29 +02:00
2fdf24495b
Restructure crawling and auth related modules
2021-05-23 19:16:42 +02:00
