Commit Graph

34 Commits

Author SHA1 Message Date
Joscha
fc31100a0f Always use '/' as path separator for regex rules
Previously, regex-matching paths on windows would, in some cases, require four
backslashes ('\\\\') to escape a single path separator. That's just too much.

With this commit, regex transforms now use '/' instead of '\' as path separator,
meaning rules can more easily be shared between platforms (although they are not
guaranteed to be 100% compatible since on Windows, '\' is still recognized as a
path separator).

To make rules more intuitive to write, local relative paths are now also printed
with '/' as path separator on Windows. Since Windows also accepts '/' as path
separator, this change doesn't really affect other rules that parse their sides
as paths.
2021-06-04 18:12:45 +02:00
Joscha
803e5628a2 Clean up logging
Paths are now (hopefully) logged consistently across all crawlers
2021-05-23 11:37:19 +02:00
Joscha
552cd82802 Run async input and password getters in daemon thread
Previously, it ran in the event loop's default executor, which would block until
all its workers were done working.

If Ctrl+C was pressed while input or a password were being read, the
asyncio.run() call in the main thread would be interrupted however, not the
input thread. This meant that multiple key presses (either enter or a second
Ctrl+C) were necessary to stop a running PFERD in some circumstances.

This change instead runs the input functions in daemon threads so they exit as
soon as the main thread exits.
2021-05-22 18:37:53 +02:00
I-Al-Istannen
9f03702e69 Split up ilias crawler in multiple files
The ilias crawler contained a crawler and an HTML parser, now they are
split in two.
2021-05-19 21:34:36 +02:00
Joscha
a7c025fd86 Implement reusable FileSinkToken for OutputDirectory 2021-05-19 17:16:23 +02:00
Joscha
0bae009189 Run formatting tools 2021-05-16 14:32:53 +02:00
I-Al-Istannen
b2a2b5999b Implement ILIAS auth and crawl home page
This commit introduces the necessary machinery to authenticate with
ILIAS and crawl the home page.

It can't do much yet and just silently fetches the homepage.
2021-05-15 15:25:05 +02:00
Joscha
502654d853 Fix mypy errors 2021-04-29 15:47:52 +02:00
Joscha
d96a361325 Test and fix exclusive output 2021-04-29 15:27:16 +02:00
Joscha
fbebc46c58 Load and dump config 2021-04-29 09:51:50 +02:00
I-Al-Istannen
56f2394001 Add a download progress bar 2020-05-08 17:09:56 +02:00
I-Al-Istannen
8a42a2a396 Move logging into its own file 2020-04-25 20:02:01 +02:00
Joscha
1aaa6e7ab5 Use PathLike everywhere 2020-04-24 18:41:14 +00:00
Joscha
292e516297 Change crawler and downloader output 2020-04-24 18:24:44 +00:00
Joscha
4e7333b396 Allow specifying paths as strings in Pferd 2020-04-24 11:50:40 +00:00
Joscha
1ef85c45e5 Switch Transform to PurePath 2020-04-23 17:40:43 +00:00
Joscha
5ef5a56e69 Extract Location into separate file 2020-04-23 17:38:28 +00:00
I-Al-Istannen
13bc78c889 Display reason for ignoring an element in ilias crawler 2020-04-23 13:54:58 +02:00
I-Al-Istannen
bef210ae77 Rename and implement IliasDirectoryFilter 2020-04-23 12:35:18 +02:00
Joscha
df0eb84a44 Fix TmpDir and Location
TmpDir: Clean up before and after, not just after
Location: Resolve path so that parent check works properly
2020-04-23 09:50:32 +00:00
Joscha
4ef0ffe3bf Listen to pylint and mypy 2020-04-20 17:44:58 +00:00
Joscha
6407190ae0 Soupify requests responses properly 2020-04-20 16:38:30 +00:00
Joscha
154d6b29dd Listen to pylint 2020-04-20 15:16:22 +00:00
I-Al-Istannen
135a8dce4b Fix resolve_path allowing paths outside its folder
This happened if the directory name was a prefix of the offending file name.
2020-04-20 16:07:14 +02:00
I-Al-Istannen
6584d6a905 Elaborate accept_file in new_organizer 2020-04-20 15:40:07 +02:00
I-Al-Istannen
5c2ff14839 Add "prompt_yes_no" to utils 2020-04-20 14:29:48 +02:00
Joscha
d5dd5aac06 Fix some mypy errors 2020-04-20 01:54:47 +00:00
Joscha
25043a4aaa Remove unnecessary files
Also document some plans for the new program structure in REWRITE.md
2020-04-19 19:49:43 +00:00
I-Al-Istannen
67da4e69fa Add colorful log output
Highlight the important operations (new, modified) in different colours.
2019-06-07 13:28:55 +02:00
Joscha
9bae030186 Move ilias stuff from aiohttp to requests 2019-04-25 18:52:48 +00:00
Joscha
5a1bf2188b Switch from tabs to spaces 2019-04-24 12:34:20 +00:00
Joscha
8b1a34233a Add and use utility functions for changing paths
This fixes a small bug in the example config, where some files were
put in the wrong locations.
2018-11-27 08:52:27 +00:00
Joscha
34da5d4d19 Sync files from ILIAS 2018-11-26 13:39:06 +00:00
Joscha
2034c9d426 Add FfM (Fachschaft für Mathematik) synchronizer
This commit moves exceptions and some other things into utils.py and
renames files according to python's file naming guides (kinda).

It also adds a new example config using the new FfM downloader.
2018-11-24 08:27:33 +00:00