Bump version to 3.7.0

Handle groups in cards
Add doctype header to forum threads
2025-07-12 06:02:31 +02:00 · 2024-11-13 20:38:27 +01:00 · 2024-11-11 12:53:08 +01:00 · 2024-11-05 18:36:21 +01:00 · 2024-11-05 18:30:34 +01:00 · 2024-11-04 23:53:48 +01:00
46 changed files with 3519 additions and 864 deletions
--- a/.github/dependabot.yml
+++ b/.github/dependabot.yml
@ -0,0 +1,10 @@
+version: 2
+updates:
+  - package-ecosystem: github-actions
+    directory: /
+    schedule:
+      interval: monthly
+    groups:
+      gh-actions:
+        patterns:
+          - "*"
--- a/.github/workflows/build-and-release.yml
+++ b/.github/workflows/build-and-release.yml
@ -1,6 +1,6 @@
 name: build-and-release

-on: push
+on: [push, pull_request]

 defaults:
  run:
@ -13,13 +13,12 @@ jobs:
    strategy:
      fail-fast: false
      matrix:
-        os: [ubuntu-latest, windows-latest, macos-latest]
-        python: ["3.8"]
+        os: [ubuntu-latest, windows-latest, macos-13, macos-latest]
+        python: ["3.9"]
    steps:
+      - uses: actions/checkout@v4

-      - uses: actions/checkout@v2
-
-      - uses: actions/setup-python@v2
+      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python }}

@ -34,7 +33,12 @@ jobs:
        run: ./scripts/setup --no-pip

      - name: Run checks
-        run: ./scripts/check
+        run: |
+          ./scripts/check
+          ./scripts/format
+
+      - name: Assert no changes
+        run: git diff --exit-code

      - name: Build
        run: ./scripts/build
@ -45,9 +49,9 @@ jobs:
        run: mv dist/pferd* dist/pferd-${{ matrix.os }}

      - name: Upload binary
-        uses: actions/upload-artifact@v2
+        uses: actions/upload-artifact@v4
        with:
-          name: Binaries
+          name: pferd-${{ matrix.os }}
          path: dist/pferd-${{ matrix.os }}

  release:
@ -57,18 +61,20 @@ jobs:
    steps:

      - name: Download binaries
-        uses: actions/download-artifact@v2
+        uses: actions/download-artifact@v4
        with:
-          name: Binaries
+          pattern: pferd-*
+          merge-multiple: true

      - name: Rename binaries
        run: |
          mv pferd-ubuntu-latest pferd-linux
          mv pferd-windows-latest pferd-windows.exe
+          mv pferd-macos-13 pferd-mac-x86_64
          mv pferd-macos-latest pferd-mac

      - name: Create release
-        uses: softprops/action-gh-release@v1
+        uses: softprops/action-gh-release@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
@ -76,3 +82,4 @@ jobs:
            pferd-linux
            pferd-windows.exe
            pferd-mac
+            pferd-mac-x86_64
--- a/.gitignore
+++ b/.gitignore
@ -3,6 +3,7 @@
 /PFERD.egg-info/
 __pycache__/
 /.vscode/
+/.idea/

 # pyinstaller
 /pferd.spec
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -22,6 +22,175 @@ ambiguous situations.

 ## Unreleased

+## 3.7.0 - 2024-11-13
+
+### Added
+- Support for MOB videos in page descriptions
+- Clickable links in the report to directly open new/modified/not-deleted files
+- Support for non KIT shibboleth login
+
+### Changed
+- Remove videos from description pages
+- Perform ILIAS cycle detection after processing the transform to allow
+  ignoring duplicated elements
+- Parse headings (h1-h3) as folders in kit-ipd crawler
+
+### Fixed
+- Personal desktop/dashboard/favorites crawling
+- Crawling of nested courses
+- Downloading of links with no target URL
+- Handle row flex on description pages
+- Add `<!DOCTYPE html>` heading to forum threads to fix mime type detection
+- Handle groups in cards
+
+## 3.6.0 - 2024-10-23
+
+### Added
+- Generic `ilias-web` crawler and `ilias-web` CLI command
+- Support for the course overview page. Using this URL as a target might cause
+  duplication warnings, as subgroups are listed separately.
+- Support for named capture groups in regex transforms
+- Crawl custom item groups as folders
+
+### Fixed
+- Normalization of meeting names in cards
+- Sanitization of slashes in exercise container names
+
+## 3.5.2 - 2024-04-14
+
+### Fixed
+- Crawling of personal desktop with ILIAS 8
+- Crawling of empty personal desktops
+
+## 3.5.1 - 2024-04-09
+
+### Added
+- Support for ILIAS 8
+
+### Fixed
+- Video name deduplication
+
+## 3.5.0 - 2023-09-13
+
+### Added
+- `no-delete-prompt-override` conflict resolution strategy
+- Support for ILIAS learning modules
+- `show_not_deleted` option to stop printing the "Not Deleted" status or report
+  message. This combines nicely with the `no-delete-prompt-override` strategy,
+  causing PFERD to mostly ignore local-only files.
+- Support for mediacast video listings
+- Crawling of files in info tab
+
+### Changed
+- Remove size suffix for files in content pages
+
+### Fixed
+- Crawling of courses with the timeline view as the default tab
+- Crawling of file and custom opencast cards
+- Crawling of button cards without descriptions
+- Abort crawling when encountering an unexpected ilias root page redirect
+- Sanitize ascii control characters on Windows
+- Crawling of paginated past meetings
+- Ignore SCORM learning modules
+
+## 3.4.3 - 2022-11-29
+
+### Added
+- Missing documentation for `forums` option
+
+### Changed
+- Clear up error message shown when multiple paths are found to an element
+
+### Fixed
+- IPD crawler unnecessarily appending trailing slashes
+- Crawling opencast when ILIAS is set to English
+
+## 3.4.2 - 2022-10-26
+
+### Added
+- Recognize and crawl content pages in cards
+- Recognize and ignore surveys
+
+### Fixed
+- Forum crawling crashing when a thread has no messages at all
+- Forum crawling crashing when a forum has no threads at all
+- Ilias login failing in some cases
+- Crawling of paginated future meetings
+- IPD crawler handling of URLs without trailing slash
+
+## 3.4.1 - 2022-08-17
+
+### Added
+- Download of page descriptions
+- Forum download support
+- `pass` authenticator
+
+### Changed
+- Add `cpp` extension to default `link_regex` of IPD crawler
+- Mention hrefs in IPD crawler's `--explain` output for users of `link_regex` option
+- Simplify default IPD crawler `link_regex`
+
+### Fixed
+- IPD crawler crashes on some sites
+- Meeting name normalization for yesterday, today and tomorrow
+- Crawling of meeting file previews
+- Login with new login button html layout
+- Descriptions for courses are now placed in the correct subfolder when
+  downloading the whole desktop
+
+## 3.4.0 - 2022-05-01
+
+### Added
+- Message when Shibboleth entitlements need to be manually reviewed
+- Links to unofficial packages and repology in the readme
+
+### Changed
+- Increase minimum supported Python version to 3.9
+- Support video listings with more columns
+- Use UTF-8 when reading/writing the config file
+
+### Fixed
+- Crash during authentication when the Shibboleth session is still valid
+
+## 3.3.1 - 2022-01-15
+
+### Fixed
+- ILIAS login
+- Local video cache if `windows_paths` is enabled
+
+## 3.3.0 - 2022-01-09
+
+### Added
+- A KIT IPD crawler
+- Support for ILIAS cards
+- (Rudimentary) support for content pages
+- Support for multi-stream videos
+- Support for ILIAS 7
+
+### Removed
+- [Interpolation](https://docs.python.org/3/library/configparser.html#interpolation-of-values) in config file
+
+### Fixed
+- Crawling of recursive courses
+- Crawling files directly placed on the personal desktop
+- Ignore timestamps at the unix epoch as they crash on windows
+
+## 3.2.0 - 2021-08-04
+
+### Added
+- `--skip` command line option
+- Support for ILIAS booking objects
+
+### Changed
+- Using multiple path segments on left side of `-name->` now results in an
+  error. This was already forbidden by the documentation but silently accepted
+  by PFERD.
+- More consistent path printing in some `--explain` messages
+
+### Fixed
+- Nondeterministic name deduplication due to ILIAS reordering elements
+- More exceptions are handled properly
+
 ## 3.1.0 - 2021-06-13

 If your config file doesn't do weird things with transforms, it should continue
--- a/CONFIG.md
+++ b/CONFIG.md
@ -4,11 +4,11 @@ A config file consists of sections. A section begins with a `[section]` header,
 which is followed by a list of `key = value` pairs. Comments must be on their
 own line and start with `#`. Multiline values must be indented beyond their key.
 Boolean values can be `yes` or `no`. For more details and some examples on the
-format, see the [configparser documentation][1] ([basic interpolation][2] is
-enabled).
+format, see the [configparser documentation][cp-file]
+([interpolation][cp-interp] is disabled).

-[1]: <https://docs.python.org/3/library/configparser.html#supported-ini-file-structure> "Supported INI File Structure"
-[2]: <https://docs.python.org/3/library/configparser.html#configparser.BasicInterpolation> "BasicInterpolation"
+[cp-file]: <https://docs.python.org/3/library/configparser.html#supported-ini-file-structure> "Supported INI File Structure"
+[cp-interp]: <https://docs.python.org/3/library/configparser.html#interpolation-of-values> "Interpolation of values"

 ## The `DEFAULT` section

@ -26,6 +26,9 @@ default values for the other sections.
  `Added ...`) while running a crawler. (Default: `yes`)
 - `report`: Whether PFERD should print a report of added, changed and deleted
   local files for all crawlers before exiting. (Default: `yes`)
+- `show_not_deleted`: Whether PFERD should print messages in status and report
+   when a local-only file wasn't deleted. Combines nicely with the
+   `no-delete-prompt-override` conflict resolution strategy.
 - `share_cookies`: Whether crawlers should share cookies where applicable. For
  example, some crawlers share cookies if they crawl the same website using the
  same account. (Default: `yes`)
@ -36,7 +39,7 @@ Sections whose names start with `crawl:` are used to configure crawlers. The
 rest of the section name specifies the name of the crawler.

 A crawler synchronizes a remote resource to a local directory. There are
-different types of crawlers for different kinds of resources, e. g. ILIAS
+different types of crawlers for different kinds of resources, e.g. ILIAS
 courses or lecture websites.

 Each crawl section represents an instance of a specific type of crawler. The
@ -53,7 +56,7 @@ common to all crawlers:
  crawler can still be executed manually using the `--crawler` or `-C` flags.
  (Default: `no`)
 - `output_dir`: The directory the crawler synchronizes files to. A crawler will
-  never place any files outside of this directory. (Default: the crawler's name)
+  never place any files outside this directory. (Default: the crawler's name)
 - `redownload`: When to download a file that is already present locally.
  (Default: `never-smart`)
    - `never`: If a file is present locally, it is not downloaded again.
@ -75,6 +78,9 @@ common to all crawlers:
      using `prompt` and always choosing "yes".
    - `no-delete`: Never delete local files, but overwrite local files if the
      remote file is different.
+    - `no-delete-prompt-overwrite`: Never delete local files, but prompt to
+      overwrite local files if the remote file is different. Combines nicely
+      with the `show_not_deleted` option.
 - `transform`: Rules for renaming and excluding certain files and directories.
  For more details, see [this section](#transformation-rules). (Default: empty)
 - `tasks`: The maximum number of concurrent tasks (such as crawling or
@ -136,6 +142,75 @@ crawler simulate a slower, network-based crawler.
  requests. (Default: `0.0`)
 - `download_speed`: Download speed (in bytes per second) to simulate. (Optional)

+### The `kit-ipd` crawler
+
+This crawler crawls a KIT-IPD page by url. The root page can be crawled from
+outside the KIT network so you will be informed about any new/deleted files,
+but downloading files requires you to be within. Adding a short delay between
+requests is likely a good idea.
+
+- `target`: URL to a KIT-IPD page
+- `link_regex`: A regex that is matched against the `href` part of links. If it
+  matches, the given link is downloaded as a file. This is used to extract
+  files from KIT-IPD pages. (Default: `^.*?[^/]+\.(pdf|zip|c|cpp|java)$`)
+
+### The `ilias-web` crawler
+
+This crawler crawls a generic ILIAS instance.
+
+Inspired by [this ILIAS downloader][ilias-dl], the following configurations should work
+out of the box for the corresponding universities:
+
+[ilias-dl]: https://github.com/V3lop5/ilias-downloader/blob/main/configs "ilias-downloader configs"
+
+| University    | `base_url`                              | `login_type` | `client_id`   |
+|---------------|-----------------------------------------|--------------|---------------|
+| FH Aachen     | https://www.ili.fh-aachen.de            | local        | elearning     |
+| Uni Köln      | https://www.ilias.uni-koeln.de/ilias    | local        | uk            |
+| Uni Konstanz  | https://ilias.uni-konstanz.de           | local        | ILIASKONSTANZ |
+| Uni Stuttgart | https://ilias3.uni-stuttgart.de         | local        | Uni_Stuttgart |
+| Uni Tübingen  | https://ovidius.uni-tuebingen.de/ilias3 | shibboleth   |               |
+
+If your university isn't listed, try navigating to your instance's login page.
+Assuming no custom login service is used, the URL will look something like this:
+
+```jinja
+{{ base_url }}/login.php?client_id={{ client_id }}&cmd=force_login&lang=
+```
+
+If the values work, feel free to submit a PR and add them to the table above.
+
+- `base_url`: The URL where the ILIAS instance is located. (Required)
+- `login_type`: How you authenticate. (Required)
+    - `local`: Use `client_id` for authentication.
+    - `shibboleth`: Use shibboleth for authentication.
+- `client_id`: An ID used for authentication if `login_type` is `local`. Is
+  ignored if `login_type` is `shibboleth`.
+- `target`: The ILIAS element to crawl. (Required)
+    - `desktop`: Crawl your personal desktop / dashboard
+    - `<course id>`: Crawl the course with the given id
+    - `<url>`: Crawl a given element by URL (preferably the permanent URL linked
+      at the bottom of its ILIAS page).  
+      This also supports the "My Courses" overview page to download *all*
+      courses. Note that this might produce confusing local directory layouts
+      and duplication warnings if you are a member of an ILIAS group. The
+      `desktop` target is generally preferable.
+- `auth`: Name of auth section to use for login. (Required)
+- `tfa_auth`: Name of auth section to use for two-factor authentication. Only
+  uses the auth section's password. (Default: Anonymous `tfa` authenticator)
+- `links`: How to represent external links. (Default: `fancy`)
+    - `ignore`: Don't download links.
+    - `plaintext`: A text file containing only the URL.
+    - `fancy`: A HTML file looking like the ILIAS link element.
+    - `internet-shortcut`: An internet shortcut file (`.url` file).
+- `link_redirect_delay`: Time (in seconds) until `fancy` link files will
+  redirect to the actual URL. Set to a negative value to disable the automatic
+  redirect. (Default: `-1`)
+- `videos`: Whether to download videos. (Default: `no`)
+- `forums`: Whether to download forum threads. (Default: `no`)
+- `http_timeout`: The timeout (in seconds) for all HTTP requests. (Default:
+  `20.0`)
+
 ### The `kit-ilias-web` crawler

 This crawler crawls the KIT ILIAS instance.
@ -169,6 +244,7 @@ script once per day should be fine.
  redirect to the actual URL. Set to a negative value to disable the automatic
  redirect. (Default: `-1`)
 - `videos`: Whether to download videos. (Default: `no`)
+- `forums`: Whether to download forum threads. (Default: `no`)
 - `http_timeout`: The timeout (in seconds) for all HTTP requests. (Default:
  `20.0`)

@ -211,6 +287,23 @@ is stored in the keyring.
 - `keyring_name`: The service name PFERD uses for storing credentials. (Default:
  `PFERD`)

+### The `pass` authenticator
+
+This authenticator queries the [`pass` password manager][pass] for a username
+and password. It tries to be mostly compatible with [browserpass][browserpass]
+and [passff][passff], so see those links for an overview of the format. If PFERD
+fails to load your password, you can use the `--explain` flag to see why.
+
+- `passname`: The name of the password to use (Required)
+- `username_prefixes`: A comma-separated list of username line prefixes
+  (Default: `login,username,user`)
+- `password_prefixes`: A comma-separated list of password line prefixes
+  (Default: `password,pass,secret`)
+
+[pass]: <https://www.passwordstore.org/> "Pass: The Standard Unix Password Manager"
+[browserpass]: <https://github.com/browserpass/browserpass-extension#organizing-password-store> "Organizing password store"
+[passff]: <https://github.com/passff/passff#multi-line-format> "Multi-line format"
+
 ### The `tfa` authenticator

 This authenticator prompts the user on the console for a two-factor
@ -260,7 +353,7 @@ path matches `SOURCE`, it is renamed to `TARGET`.
 Example: `foo/bar --> baz`
 - Doesn't match `foo`, `a/foo/bar` or `foo/baz`
 - Converts `foo/bar` into `baz`
- Converts `foo/bar/wargl` into `bar/wargl`
+- Converts `foo/bar/wargl` into `baz/wargl`

 Example: `foo/bar --> !`
 - Doesn't match `foo`, `a/foo/bar` or `foo/baz`
@ -304,12 +397,13 @@ is a regular expression and `TARGET` an f-string based template. If a path
 matches `SOURCE`, the output path is created using `TARGET` as template.
 `SOURCE` is automatically anchored.

-`TARGET` uses Python's [format string syntax][3]. The *n*-th capturing group can
-be referred to as `{g<n>}` (e. g. `{g3}`). `{g0}` refers to the original path.
+`TARGET` uses Python's [format string syntax][6]. The *n*-th capturing group can
+be referred to as `{g<n>}` (e.g. `{g3}`). `{g0}` refers to the original path.
 If capturing group *n*'s contents are a valid integer, the integer value is
-available as `{i<n>}` (e. g. `{i3}`). If capturing group *n*'s contents are a
-valid float, the float value is available as `{f<n>}` (e. g. `{f3}`). If a
-capturing group is not present (e. g. when matching the string `cd` with the
+available as `{i<n>}` (e.g. `{i3}`). If capturing group *n*'s contents are a
+valid float, the float value is available as `{f<n>}` (e.g. `{f3}`). Named capture
+groups (e.g. `(?P<name>)`) are available by their name (e.g. `{name}`). If a
+capturing group is not present (e.g. when matching the string `cd` with the
 regex `(ab)?cd`), the corresponding variables are not defined.

 Python's format string syntax has rich options for formatting its arguments. For
@ -325,7 +419,7 @@ Example: `f(oo+)/be?ar -re-> B{g1.upper()}H/fear`
 - Converts `fooooo/bear` into `BOOOOOH/fear`
 - Converts `foo/bar/baz` into `BOOH/fear/baz`

-[3]: <https://docs.python.org/3/library/string.html#format-string-syntax> "Format String Syntax"
+[6]: <https://docs.python.org/3/library/string.html#format-string-syntax> "Format String Syntax"

 ### The `-name-re->` arrow

--- a/4
+++ b/4
@ -1,4 +1,6 @@
-Copyright 2019-2020 Garmelon, I-Al-Istannen, danstooamerican, pavelzw, TheChristophe, Scriptim
+Copyright 2019-2024 Garmelon, I-Al-Istannen, danstooamerican, pavelzw,
+                    TheChristophe, Scriptim, thelukasprobst, Toorero,
+                    Mr-Pine, p-fruck, PinieP

 Permission is hereby granted, free of charge, to any person obtaining a copy of
 this software and associated documentation files (the "Software"), to deal in
--- a/PFERD/main.py
+++ b/PFERD/main.py
@ -15,7 +15,7 @@ from .transformer import RuleParseError

 def load_config_parser(args: argparse.Namespace) -> configparser.ConfigParser:
    log.explain_topic("Loading config")
-    parser = configparser.ConfigParser()
+    parser = configparser.ConfigParser(interpolation=None)

    if args.command is None:
        log.explain("No CLI command specified, loading config from file")
@ -47,6 +47,8 @@ def configure_logging_from_args(args: argparse.Namespace) -> None:
        log.output_explain = args.explain
    if args.status is not None:
        log.output_status = args.status
+    if args.show_not_deleted is not None:
+        log.output_not_deleted = args.show_not_deleted
    if args.report is not None:
        log.output_report = args.report

@ -72,6 +74,8 @@ def configure_logging_from_config(args: argparse.Namespace, config: Config) -> N
            log.output_status = config.default_section.status()
        if args.report is None:
            log.output_report = config.default_section.report()
+        if args.show_not_deleted is None:
+            log.output_not_deleted = config.default_section.show_not_deleted()
    except ConfigOptionError as e:
        log.error(str(e))
        sys.exit(1)
@ -116,7 +120,7 @@ def main() -> None:
        sys.exit()

    try:
-        pferd = Pferd(config, args.crawler)
+        pferd = Pferd(config, args.crawler, args.skip)
    except PferdLoadError as e:
        log.unlock()
        log.error(str(e))
@ -159,3 +163,7 @@ def main() -> None:
        sys.exit(1)
    else:
        pferd.print_report()
+
+
+if __name__ == "__main__":
+    main()
--- a/PFERD/auth/init.py
+++ b/PFERD/auth/init.py
@ -5,6 +5,7 @@ from ..config import Config
 from .authenticator import Authenticator, AuthError, AuthLoadError, AuthSection  # noqa: F401
 from .credential_file import CredentialFileAuthenticator, CredentialFileAuthSection
 from .keyring import KeyringAuthenticator, KeyringAuthSection
+from .pass_ import PassAuthenticator, PassAuthSection
 from .simple import SimpleAuthenticator, SimpleAuthSection
 from .tfa import TfaAuthenticator

@ -19,6 +20,8 @@ AUTHENTICATORS: Dict[str, AuthConstructor] = {
        CredentialFileAuthenticator(n, CredentialFileAuthSection(s), c),
    "keyring": lambda n, s, c:
        KeyringAuthenticator(n, KeyringAuthSection(s)),
+    "pass": lambda n, s, c:
+        PassAuthenticator(n, PassAuthSection(s)),
    "simple": lambda n, s, c:
        SimpleAuthenticator(n, SimpleAuthSection(s)),
    "tfa": lambda n, s, c:
--- a/PFERD/auth/credential_file.py
+++ b/PFERD/auth/credential_file.py
@ -20,8 +20,10 @@ class CredentialFileAuthenticator(Authenticator):

        path = config.default_section.working_dir() / section.path()
        try:
-            with open(path) as f:
+            with open(path, encoding="utf-8") as f:
                lines = list(f)
+        except UnicodeDecodeError:
+            raise AuthLoadError(f"Credential file at {fmt_real_path(path)} is not encoded using UTF-8")
        except OSError as e:
            raise AuthLoadError(f"No credential file at {fmt_real_path(path)}") from e

--- a/PFERD/auth/pass_.py
+++ b/PFERD/auth/pass_.py
@ -0,0 +1,98 @@
+import re
+import subprocess
+from typing import List, Tuple
+
+from ..logging import log
+from .authenticator import Authenticator, AuthError, AuthSection
+
+
+class PassAuthSection(AuthSection):
+    def passname(self) -> str:
+        if (value := self.s.get("passname")) is None:
+            self.missing_value("passname")
+        return value
+
+    def username_prefixes(self) -> List[str]:
+        value = self.s.get("username_prefixes", "login,username,user")
+        return [prefix.lower() for prefix in value.split(",")]
+
+    def password_prefixes(self) -> List[str]:
+        value = self.s.get("password_prefixes", "password,pass,secret")
+        return [prefix.lower() for prefix in value.split(",")]
+
+
+class PassAuthenticator(Authenticator):
+    PREFIXED_LINE_RE = r"([a-zA-Z]+):\s?(.*)"  # to be used with fullmatch
+
+    def __init__(self, name: str, section: PassAuthSection) -> None:
+        super().__init__(name)
+
+        self._passname = section.passname()
+        self._username_prefixes = section.username_prefixes()
+        self._password_prefixes = section.password_prefixes()
+
+    async def credentials(self) -> Tuple[str, str]:
+        log.explain_topic("Obtaining credentials from pass")
+
+        try:
+            log.explain(f"Calling 'pass show {self._passname}'")
+            result = subprocess.check_output(["pass", "show", self._passname], text=True)
+        except subprocess.CalledProcessError as e:
+            raise AuthError(f"Failed to get password info from {self._passname}: {e}")
+
+        prefixed = {}
+        unprefixed = []
+        for line in result.strip().splitlines():
+            if match := re.fullmatch(self.PREFIXED_LINE_RE, line):
+                prefix = match.group(1).lower()
+                value = match.group(2)
+                log.explain(f"Found prefixed line {line!r} with prefix {prefix!r}, value {value!r}")
+                if prefix in prefixed:
+                    raise AuthError(f"Prefix {prefix} specified multiple times")
+                prefixed[prefix] = value
+            else:
+                log.explain(f"Found unprefixed line {line!r}")
+                unprefixed.append(line)
+
+        username = None
+        for prefix in self._username_prefixes:
+            log.explain(f"Looking for username at prefix {prefix!r}")
+            if prefix in prefixed:
+                username = prefixed[prefix]
+                log.explain(f"Found username {username!r}")
+                break
+
+        password = None
+        for prefix in self._password_prefixes:
+            log.explain(f"Looking for password at prefix {prefix!r}")
+            if prefix in prefixed:
+                password = prefixed[prefix]
+                log.explain(f"Found password {password!r}")
+                break
+
+        if password is None and username is None:
+            log.explain("No username and password found so far")
+            log.explain("Using first unprefixed line as password")
+            log.explain("Using second unprefixed line as username")
+        elif password is None:
+            log.explain("No password found so far")
+            log.explain("Using first unprefixed line as password")
+        elif username is None:
+            log.explain("No username found so far")
+            log.explain("Using first unprefixed line as username")
+
+        if password is None:
+            if not unprefixed:
+                log.explain("Not enough unprefixed lines left")
+                raise AuthError("Password could not be determined")
+            password = unprefixed.pop(0)
+            log.explain(f"Found password {password!r}")
+
+        if username is None:
+            if not unprefixed:
+                log.explain("Not enough unprefixed lines left")
+                raise AuthError("Username could not be determined")
+            username = unprefixed.pop(0)
+            log.explain(f"Found username {username!r}")
+
+        return username, password
--- a/PFERD/cli/init.py
+++ b/PFERD/cli/init.py
@ -8,5 +8,7 @@
 # well.

 from . import command_local  # noqa: F401 imported but unused
+from . import command_ilias_web  # noqa: F401 imported but unused
 from . import command_kit_ilias_web  # noqa: F401 imported but unused
+from . import command_kit_ipd  # noqa: F401 imported but unused
 from .parser import PARSER, ParserLoadError, load_default_section  # noqa: F401 imported but unused
--- a/PFERD/cli/command_ilias_web.py
+++ b/PFERD/cli/command_ilias_web.py
@ -0,0 +1,56 @@
+import argparse
+import configparser
+
+from ..logging import log
+from .common_ilias_args import configure_common_group_args, load_common
+from .parser import CRAWLER_PARSER, SUBPARSERS, load_crawler
+
+COMMAND_NAME = "ilias-web"
+
+SUBPARSER = SUBPARSERS.add_parser(
+    COMMAND_NAME,
+    parents=[CRAWLER_PARSER],
+)
+
+GROUP = SUBPARSER.add_argument_group(
+    title=f"{COMMAND_NAME} crawler arguments",
+    description=f"arguments for the '{COMMAND_NAME}' crawler",
+)
+
+GROUP.add_argument(
+    "--base-url",
+    type=str,
+    metavar="BASE_URL",
+    help="The base url of the ilias instance"
+)
+
+GROUP.add_argument(
+    "--client-id",
+    type=str,
+    metavar="CLIENT_ID",
+    help="The client id of the ilias instance"
+)
+
+configure_common_group_args(GROUP)
+
+
+def load(
+        args: argparse.Namespace,
+        parser: configparser.ConfigParser,
+) -> None:
+    log.explain(f"Creating config for command '{COMMAND_NAME}'")
+
+    parser["crawl:ilias"] = {}
+    section = parser["crawl:ilias"]
+    load_crawler(args, section)
+
+    section["type"] = COMMAND_NAME
+    if args.ilias_url is not None:
+        section["base_url"] = args.ilias_url
+    if args.client_id is not None:
+        section["client_id"] = args.client_id
+
+    load_common(section, args, parser)
+
+
+SUBPARSER.set_defaults(command=load)
--- a/PFERD/cli/command_kit_ilias_web.py
+++ b/PFERD/cli/command_kit_ilias_web.py
@ -1,113 +1,37 @@
 import argparse
 import configparser
-from pathlib import Path

-from ..crawl.ilias.file_templates import Links
 from ..logging import log
-from .parser import (CRAWLER_PARSER, SUBPARSERS, BooleanOptionalAction, ParserLoadError, load_crawler,
-                     show_value_error)
+from .common_ilias_args import configure_common_group_args, load_common
+from .parser import CRAWLER_PARSER, SUBPARSERS, load_crawler
+
+COMMAND_NAME = "kit-ilias-web"

 SUBPARSER = SUBPARSERS.add_parser(
-    "kit-ilias-web",
+    COMMAND_NAME,
    parents=[CRAWLER_PARSER],
 )

 GROUP = SUBPARSER.add_argument_group(
-    title="kit-ilias-web crawler arguments",
-    description="arguments for the 'kit-ilias-web' crawler",
-)
-GROUP.add_argument(
-    "target",
-    type=str,
-    metavar="TARGET",
-    help="course id, 'desktop', or ILIAS URL to crawl"
-)
-GROUP.add_argument(
-    "output",
-    type=Path,
-    metavar="OUTPUT",
-    help="output directory"
-)
-GROUP.add_argument(
-    "--username", "-u",
-    type=str,
-    metavar="USERNAME",
-    help="user name for authentication"
-)
-GROUP.add_argument(
-    "--keyring",
-    action=BooleanOptionalAction,
-    help="use the system keyring to store and retrieve passwords"
-)
-GROUP.add_argument(
-    "--credential-file",
-    type=Path,
-    metavar="PATH",
-    help="read username and password from a credential file"
-)
-GROUP.add_argument(
-    "--links",
-    type=show_value_error(Links.from_string),
-    metavar="OPTION",
-    help="how to represent external links"
-)
-GROUP.add_argument(
-    "--link-redirect-delay",
-    type=int,
-    metavar="SECONDS",
-    help="time before 'fancy' links redirect to to their target (-1 to disable)"
-)
-GROUP.add_argument(
-    "--videos",
-    action=BooleanOptionalAction,
-    help="crawl and download videos"
-)
-GROUP.add_argument(
-    "--http-timeout", "-t",
-    type=float,
-    metavar="SECONDS",
-    help="timeout for all HTTP requests"
+    title=f"{COMMAND_NAME} crawler arguments",
+    description=f"arguments for the '{COMMAND_NAME}' crawler",
 )

+configure_common_group_args(GROUP)
+

 def load(
        args: argparse.Namespace,
        parser: configparser.ConfigParser,
 ) -> None:
-    log.explain("Creating config for command 'kit-ilias-web'")
+    log.explain(f"Creating config for command '{COMMAND_NAME}'")

    parser["crawl:ilias"] = {}
    section = parser["crawl:ilias"]
    load_crawler(args, section)

-    section["type"] = "kit-ilias-web"
-    section["target"] = str(args.target)
-    section["output_dir"] = str(args.output)
-    section["auth"] = "auth:ilias"
-    if args.links is not None:
-        section["links"] = str(args.links.value)
-    if args.link_redirect_delay is not None:
-        section["link_redirect_delay"] = str(args.link_redirect_delay)
-    if args.videos is not None:
-        section["videos"] = "yes" if args.videos else "no"
-    if args.http_timeout is not None:
-        section["http_timeout"] = str(args.http_timeout)
-
-    parser["auth:ilias"] = {}
-    auth_section = parser["auth:ilias"]
-    if args.credential_file is not None:
-        if args.username is not None:
-            raise ParserLoadError("--credential-file and --username can't be used together")
-        if args.keyring:
-            raise ParserLoadError("--credential-file and --keyring can't be used together")
-        auth_section["type"] = "credential-file"
-        auth_section["path"] = str(args.credential_file)
-    elif args.keyring:
-        auth_section["type"] = "keyring"
-    else:
-        auth_section["type"] = "simple"
-    if args.username is not None:
-        auth_section["username"] = args.username
+    section["type"] = COMMAND_NAME
+    load_common(section, args, parser)


 SUBPARSER.set_defaults(command=load)
--- a/PFERD/cli/command_kit_ipd.py
+++ b/PFERD/cli/command_kit_ipd.py
@ -0,0 +1,54 @@
+import argparse
+import configparser
+from pathlib import Path
+
+from ..logging import log
+from .parser import CRAWLER_PARSER, SUBPARSERS, load_crawler
+
+SUBPARSER = SUBPARSERS.add_parser(
+    "kit-ipd",
+    parents=[CRAWLER_PARSER],
+)
+
+GROUP = SUBPARSER.add_argument_group(
+    title="kit ipd crawler arguments",
+    description="arguments for the 'kit-ipd' crawler",
+)
+GROUP.add_argument(
+    "--link-regex",
+    type=str,
+    metavar="REGEX",
+    help="href-matching regex to identify downloadable files"
+)
+GROUP.add_argument(
+    "target",
+    type=str,
+    metavar="TARGET",
+    help="url to crawl"
+)
+GROUP.add_argument(
+    "output",
+    type=Path,
+    metavar="OUTPUT",
+    help="output directory"
+)
+
+
+def load(
+        args: argparse.Namespace,
+        parser: configparser.ConfigParser,
+) -> None:
+    log.explain("Creating config for command 'kit-ipd'")
+
+    parser["crawl:kit-ipd"] = {}
+    section = parser["crawl:kit-ipd"]
+    load_crawler(args, section)
+
+    section["type"] = "kit-ipd"
+    section["target"] = str(args.target)
+    section["output_dir"] = str(args.output)
+    if args.link_regex:
+        section["link_regex"] = str(args.link_regex)
+
+
+SUBPARSER.set_defaults(command=load)
--- a/PFERD/cli/common_ilias_args.py
+++ b/PFERD/cli/common_ilias_args.py
@ -0,0 +1,104 @@
+import argparse
+import configparser
+from pathlib import Path
+
+from ..crawl.ilias.file_templates import Links
+from .parser import BooleanOptionalAction, ParserLoadError, show_value_error
+
+
+def configure_common_group_args(group: argparse._ArgumentGroup) -> None:
+    """These arguments are shared between the KIT and generic Ilias web command."""
+    group.add_argument(
+        "target",
+        type=str,
+        metavar="TARGET",
+        help="course id, 'desktop', or ILIAS URL to crawl"
+    )
+    group.add_argument(
+        "output",
+        type=Path,
+        metavar="OUTPUT",
+        help="output directory"
+    )
+    group.add_argument(
+        "--username", "-u",
+        type=str,
+        metavar="USERNAME",
+        help="user name for authentication"
+    )
+    group.add_argument(
+        "--keyring",
+        action=BooleanOptionalAction,
+        help="use the system keyring to store and retrieve passwords"
+    )
+    group.add_argument(
+        "--credential-file",
+        type=Path,
+        metavar="PATH",
+        help="read username and password from a credential file"
+    )
+    group.add_argument(
+        "--links",
+        type=show_value_error(Links.from_string),
+        metavar="OPTION",
+        help="how to represent external links"
+    )
+    group.add_argument(
+        "--link-redirect-delay",
+        type=int,
+        metavar="SECONDS",
+        help="time before 'fancy' links redirect to to their target (-1 to disable)"
+    )
+    group.add_argument(
+        "--videos",
+        action=BooleanOptionalAction,
+        help="crawl and download videos"
+    )
+    group.add_argument(
+        "--forums",
+        action=BooleanOptionalAction,
+        help="crawl and download forum posts"
+    )
+    group.add_argument(
+        "--http-timeout", "-t",
+        type=float,
+        metavar="SECONDS",
+        help="timeout for all HTTP requests"
+    )
+
+
+def load_common(
+    section: configparser.SectionProxy,
+    args: argparse.Namespace,
+    parser: configparser.ConfigParser,
+) -> None:
+    """Load common config between generic and KIT ilias web command"""
+    section["target"] = str(args.target)
+    section["output_dir"] = str(args.output)
+    section["auth"] = "auth:ilias"
+    if args.links is not None:
+        section["links"] = str(args.links.value)
+    if args.link_redirect_delay is not None:
+        section["link_redirect_delay"] = str(args.link_redirect_delay)
+    if args.videos is not None:
+        section["videos"] = "yes" if args.videos else "no"
+    if args.forums is not None:
+        section["forums"] = "yes" if args.forums else "no"
+    if args.http_timeout is not None:
+        section["http_timeout"] = str(args.http_timeout)
+
+    parser["auth:ilias"] = {}
+    auth_section = parser["auth:ilias"]
+    if args.credential_file is not None:
+        if args.username is not None:
+            raise ParserLoadError("--credential-file and --username can't be used together")
+        if args.keyring:
+            raise ParserLoadError("--credential-file and --keyring can't be used together")
+        auth_section["type"] = "credential-file"
+        auth_section["path"] = str(args.credential_file)
+    elif args.keyring:
+        auth_section["type"] = "keyring"
+    else:
+        auth_section["type"] = "simple"
+    if args.username is not None:
+        auth_section["username"] = args.username
--- a/PFERD/cli/parser.py
+++ b/PFERD/cli/parser.py
@ -181,6 +181,14 @@ PARSER.add_argument(
    help="only execute a single crawler."
    " Can be specified multiple times to execute multiple crawlers"
 )
+PARSER.add_argument(
+    "--skip", "-S",
+    action="append",
+    type=str,
+    metavar="NAME",
+    help="don't execute this particular crawler."
+    " Can be specified multiple times to skip multiple crawlers"
+)
 PARSER.add_argument(
    "--working-dir",
    type=Path,
@ -207,6 +215,11 @@ PARSER.add_argument(
    action=BooleanOptionalAction,
    help="whether crawlers should share cookies where applicable"
 )
+PARSER.add_argument(
+    "--show-not-deleted",
+    action=BooleanOptionalAction,
+    help="print messages in status and report when PFERD did not delete a local only file"
+)


 def load_default_section(
@ -225,6 +238,8 @@ def load_default_section(
        section["report"] = "yes" if args.report else "no"
    if args.share_cookies is not None:
        section["share_cookies"] = "yes" if args.share_cookies else "no"
+    if args.show_not_deleted is not None:
+        section["show_not_deleted"] = "yes" if args.show_not_deleted else "no"


 SUBPARSERS = PARSER.add_subparsers(title="crawlers")
--- a/PFERD/config.py
+++ b/PFERD/config.py
@ -82,6 +82,9 @@ class DefaultSection(Section):
    def report(self) -> bool:
        return self.s.getboolean("report", fallback=True)

+    def show_not_deleted(self) -> bool:
+        return self.s.getboolean("show_not_deleted", fallback=True)
+
    def share_cookies(self) -> bool:
        return self.s.getboolean("share_cookies", fallback=True)

@ -120,7 +123,7 @@ class Config:
        # Using config.read_file instead of config.read because config.read
        # would just ignore a missing file and carry on.
        try:
-            with open(path) as f:
+            with open(path, encoding="utf-8") as f:
                parser.read_file(f, source=str(path))
        except FileNotFoundError:
            raise ConfigLoadError(path, "File does not exist")
@ -128,6 +131,8 @@ class Config:
            raise ConfigLoadError(path, "That's a directory, not a file")
        except PermissionError:
            raise ConfigLoadError(path, "Insufficient permissions")
+        except UnicodeDecodeError:
+            raise ConfigLoadError(path, "File is not encoded using UTF-8")

    def dump(self, path: Optional[Path] = None) -> None:
        """
@ -154,12 +159,12 @@ class Config:
            try:
                # x = open for exclusive creation, failing if the file already
                # exists
-                with open(path, "x") as f:
+                with open(path, "x", encoding="utf-8") as f:
                    self._parser.write(f)
            except FileExistsError:
                print("That file already exists.")
                if asyncio.run(prompt_yes_no("Overwrite it?", default=False)):
-                    with open(path, "w") as f:
+                    with open(path, "w", encoding="utf-8") as f:
                        self._parser.write(f)
                else:
                    raise ConfigDumpError(path, "File already exists")
--- a/PFERD/crawl/init.py
+++ b/PFERD/crawl/init.py
@ -4,7 +4,8 @@ from typing import Callable, Dict
 from ..auth import Authenticator
 from ..config import Config
 from .crawler import Crawler, CrawlError, CrawlerSection  # noqa: F401
-from .ilias import KitIliasWebCrawler, KitIliasWebCrawlerSection
+from .ilias import IliasWebCrawler, IliasWebCrawlerSection, KitIliasWebCrawler, KitIliasWebCrawlerSection
+from .kit_ipd_crawler import KitIpdCrawler, KitIpdCrawlerSection
 from .local_crawler import LocalCrawler, LocalCrawlerSection

 CrawlerConstructor = Callable[[
@ -17,6 +18,10 @@ CrawlerConstructor = Callable[[
 CRAWLERS: Dict[str, CrawlerConstructor] = {
    "local": lambda n, s, c, a:
        LocalCrawler(n, LocalCrawlerSection(s), c),
+    "ilias-web": lambda n, s, c, a:
+        IliasWebCrawler(n, IliasWebCrawlerSection(s), c, a),
    "kit-ilias-web": lambda n, s, c, a:
        KitIliasWebCrawler(n, KitIliasWebCrawlerSection(s), c, a),
+    "kit-ipd": lambda n, s, c, a:
+        KitIpdCrawler(n, KitIpdCrawlerSection(s), c),
 }
--- a/PFERD/crawl/crawler.py
+++ b/PFERD/crawl/crawler.py
@ -1,9 +1,10 @@
 import asyncio
 import os
 from abc import ABC, abstractmethod
+from collections.abc import Awaitable, Coroutine
 from datetime import datetime
 from pathlib import Path, PurePath
-from typing import Any, Awaitable, Callable, Dict, List, Optional, Sequence, Set, Tuple, TypeVar
+from typing import Any, Callable, Dict, List, Optional, Sequence, Set, Tuple, TypeVar

 from ..auth import Authenticator
 from ..config import Config, Section
@ -47,16 +48,18 @@ def noncritical(f: Wrapped) -> Wrapped:
        try:
            f(*args, **kwargs)
        except (CrawlWarning, OutputDirError, MarkDuplicateError, MarkConflictError) as e:
+            crawler.report.add_warning(str(e))
            log.warn(str(e))
            crawler.error_free = False
-        except:  # noqa: E722 do not use bare 'except'
+        except Exception as e:
            crawler.error_free = False
+            crawler.report.add_error(str(e))
            raise

    return wrapper  # type: ignore


-AWrapped = TypeVar("AWrapped", bound=Callable[..., Awaitable[None]])
+AWrapped = TypeVar("AWrapped", bound=Callable[..., Coroutine[Any, Any, Optional[Any]]])


 def anoncritical(f: AWrapped) -> AWrapped:
@ -72,21 +75,25 @@ def anoncritical(f: AWrapped) -> AWrapped:
    Warning: Must only be applied to member functions of the Crawler class!
    """

-    async def wrapper(*args: Any, **kwargs: Any) -> None:
+    async def wrapper(*args: Any, **kwargs: Any) -> Optional[Any]:
        if not (args and isinstance(args[0], Crawler)):
            raise RuntimeError("@anoncritical must only applied to Crawler methods")

        crawler = args[0]

        try:
-            await f(*args, **kwargs)
+            return await f(*args, **kwargs)
        except (CrawlWarning, OutputDirError, MarkDuplicateError, MarkConflictError) as e:
            log.warn(str(e))
            crawler.error_free = False
-        except:  # noqa: E722 do not use bare 'except'
+            crawler.report.add_warning(str(e))
+        except Exception as e:
            crawler.error_free = False
+            crawler.report.add_error(str(e))
            raise

+        return None
+
    return wrapper  # type: ignore


@ -251,6 +258,10 @@ class Crawler(ABC):
    def prev_report(self) -> Optional[Report]:
        return self._output_dir.prev_report

+    @property
+    def output_dir(self) -> OutputDirectory:
+        return self._output_dir
+
    @staticmethod
    async def gather(awaitables: Sequence[Awaitable[Any]]) -> List[Any]:
        """
@ -286,6 +297,8 @@ class Crawler(ABC):
    async def download(
            self,
            path: PurePath,
+            *,
+            etag_differs: Optional[bool] = None,
            mtime: Optional[datetime] = None,
            redownload: Optional[Redownload] = None,
            on_conflict: Optional[OnConflict] = None,
@ -300,7 +313,14 @@ class Crawler(ABC):
            log.status("[bold bright_black]", "Ignored", fmt_path(path))
            return None

-        fs_token = await self._output_dir.download(path, transformed_path, mtime, redownload, on_conflict)
+        fs_token = await self._output_dir.download(
+            path,
+            transformed_path,
+            etag_differs=etag_differs,
+            mtime=mtime,
+            redownload=redownload,
+            on_conflict=on_conflict
+        )
        if fs_token is None:
            log.explain("Answer: No")
            return None
@ -318,6 +338,7 @@ class Crawler(ABC):
            log.explain("Warnings or errors occurred during this run")
            log.explain("Answer: No")

+    @anoncritical
    async def run(self) -> None:
        """
        Start the crawling process. Call this function if you want to use a
--- a/PFERD/crawl/http_crawler.py
+++ b/PFERD/crawl/http_crawler.py
@ -1,12 +1,14 @@
 import asyncio
 import http.cookies
 import ssl
+from datetime import datetime
 from pathlib import Path, PurePath
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, Optional, Tuple

 import aiohttp
 import certifi
 from aiohttp.client import ClientTimeout
+from bs4 import Tag

 from ..auth import Authenticator
 from ..config import Config
@ -15,6 +17,8 @@ from ..utils import fmt_real_path
 from ..version import NAME, VERSION
 from .crawler import Crawler, CrawlerSection

+ETAGS_CUSTOM_REPORT_VALUE_KEY = "etags"
+

 class HttpCrawlerSection(CrawlerSection):
    def http_timeout(self) -> float:
@ -108,7 +112,7 @@ class HttpCrawler(Crawler):

    def _load_cookies_from_file(self, path: Path) -> None:
        jar: Any = http.cookies.SimpleCookie()
-        with open(path) as f:
+        with open(path, encoding="utf-8") as f:
            for i, line in enumerate(f):
                # Names of headers are case insensitive
                if line[:11].lower() == "set-cookie:":
@ -121,7 +125,7 @@ class HttpCrawler(Crawler):
        jar: Any = http.cookies.SimpleCookie()
        for morsel in self._cookie_jar:
            jar[morsel.key] = morsel
-        with open(path, "w") as f:
+        with open(path, "w", encoding="utf-8") as f:
            f.write(jar.output(sep="\n"))
            f.write("\n")  # A trailing newline is just common courtesy

@ -169,6 +173,78 @@ class HttpCrawler(Crawler):
            log.warn(f"Failed to save cookies to {fmt_real_path(self._cookie_jar_path)}")
            log.warn(str(e))

+    @staticmethod
+    def get_folder_structure_from_heading_hierarchy(file_link: Tag, drop_h1: bool = False) -> PurePath:
+        """
+        Retrieves the hierarchy of headings associated with the give file link and constructs a folder
+        structure from them.
+
+        <h1> level headings usually only appear once and serve as the page title, so they would introduce
+        redundant nesting. To avoid this, <h1> headings are ignored via the drop_h1 parameter.
+        """
+
+        def find_associated_headings(tag: Tag, level: int) -> PurePath:
+            if level == 0 or (level == 1 and drop_h1):
+                return PurePath()
+
+            level_heading = tag.find_previous(name=f"h{level}")
+
+            if level_heading is None:
+                return find_associated_headings(tag, level - 1)
+
+            folder_name = level_heading.getText().strip()
+            return find_associated_headings(level_heading, level - 1) / folder_name
+
+        # start at level <h3> because paragraph-level headings are usually too granular for folder names
+        return find_associated_headings(file_link, 3)
+
+    def _get_previous_etag_from_report(self, path: PurePath) -> Optional[str]:
+        """
+        If available, retrieves the entity tag for a given path which was stored in the previous report.
+        """
+        if not self._output_dir.prev_report:
+            return None
+
+        etags = self._output_dir.prev_report.get_custom_value(ETAGS_CUSTOM_REPORT_VALUE_KEY) or {}
+        return etags.get(str(path))
+
+    def _add_etag_to_report(self, path: PurePath, etag: Optional[str]) -> None:
+        """
+        Adds an entity tag for a given path to the report's custom values.
+        """
+        if not etag:
+            return
+
+        etags = self._output_dir.report.get_custom_value(ETAGS_CUSTOM_REPORT_VALUE_KEY) or {}
+        etags[str(path)] = etag
+        self._output_dir.report.add_custom_value(ETAGS_CUSTOM_REPORT_VALUE_KEY, etags)
+
+    async def _request_resource_version(self, resource_url: str) -> Tuple[Optional[str], Optional[datetime]]:
+        """
+        Requests the ETag and Last-Modified headers of a resource via a HEAD request.
+        If no entity tag / modification date can be obtained, the according value will be None.
+        """
+        try:
+            async with self.session.head(resource_url) as resp:
+                if resp.status != 200:
+                    return None, None
+
+                etag_header = resp.headers.get("ETag")
+                last_modified_header = resp.headers.get("Last-Modified")
+
+                if last_modified_header:
+                    try:
+                        # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Last-Modified#directives
+                        datetime_format = "%a, %d %b %Y %H:%M:%S GMT"
+                        last_modified = datetime.strptime(last_modified_header, datetime_format)
+                    except ValueError:
+                        # last_modified remains None
+                        pass
+
+                return etag_header, last_modified
+        except aiohttp.ClientError:
+            return None, None
+
    async def run(self) -> None:
        self._request_count = 0
        self._cookie_jar = aiohttp.CookieJar()
@ -186,7 +262,12 @@ class HttpCrawler(Crawler):
                    connect=self._http_timeout,
                    sock_connect=self._http_timeout,
                    sock_read=self._http_timeout,
-                )
+                ),
+                # See https://github.com/aio-libs/aiohttp/issues/6626
+                # Without this aiohttp will mangle the redirect header from Shibboleth, invalidating the
+                # passed signature. Shibboleth will not accept the broken signature and authentication will
+                # fail.
+                requote_redirect_url=False
        ) as session:
            self.session = session
            try:
--- a/PFERD/crawl/ilias/init.py
+++ b/PFERD/crawl/ilias/init.py
@ -1,3 +1,9 @@
-from .kit_ilias_web_crawler import KitIliasWebCrawler, KitIliasWebCrawlerSection
+from .kit_ilias_web_crawler import (IliasWebCrawler, IliasWebCrawlerSection, KitIliasWebCrawler,
+                                    KitIliasWebCrawlerSection)

-__all__ = ["KitIliasWebCrawler", "KitIliasWebCrawlerSection"]
+__all__ = [
+    "IliasWebCrawler",
+    "IliasWebCrawlerSection",
+    "KitIliasWebCrawler",
+    "KitIliasWebCrawlerSection",
+]
--- a/PFERD/crawl/ilias/async_helper.py
+++ b/PFERD/crawl/ilias/async_helper.py
@ -0,0 +1,40 @@
+import asyncio
+from typing import Any, Callable, Optional
+
+import aiohttp
+
+from ...logging import log
+from ..crawler import AWrapped, CrawlError, CrawlWarning
+
+
+def _iorepeat(attempts: int, name: str, failure_is_error: bool = False) -> Callable[[AWrapped], AWrapped]:
+    def decorator(f: AWrapped) -> AWrapped:
+        async def wrapper(*args: Any, **kwargs: Any) -> Optional[Any]:
+            last_exception: Optional[BaseException] = None
+            for round in range(attempts):
+                try:
+                    return await f(*args, **kwargs)
+                except aiohttp.ContentTypeError:  # invalid content type
+                    raise CrawlWarning("ILIAS returned an invalid content type")
+                except aiohttp.TooManyRedirects:
+                    raise CrawlWarning("Got stuck in a redirect loop")
+                except aiohttp.ClientPayloadError as e:  # encoding or not enough bytes
+                    last_exception = e
+                except aiohttp.ClientConnectionError as e:  # e.g. timeout, disconnect, resolve failed, etc.
+                    last_exception = e
+                except asyncio.exceptions.TimeoutError as e:  # explicit http timeouts in HttpCrawler
+                    last_exception = e
+                log.explain_topic(f"Retrying operation {name}. Retries left: {attempts - 1 - round}")
+                log.explain(f"Last exception: {last_exception!r}")
+
+            if last_exception:
+                message = f"Error in I/O Operation: {last_exception!r}"
+                if failure_is_error:
+                    raise CrawlError(message) from last_exception
+                else:
+                    raise CrawlWarning(message) from last_exception
+            raise CrawlError("Impossible return in ilias _iorepeat")
+
+        return wrapper  # type: ignore
+
+    return decorator
--- a/PFERD/crawl/ilias/file_templates.py
+++ b/PFERD/crawl/ilias/file_templates.py
@ -1,6 +1,10 @@
 from enum import Enum
 from typing import Optional

+import bs4
+
+from PFERD.utils import soupify
+
 _link_template_plain = "{{link}}"
 _link_template_fancy = """
 <!DOCTYPE html>
@ -94,6 +98,71 @@ _link_template_internet_shortcut = """
 URL={{link}}
 """.strip()

+_learning_module_template = """
+<!DOCTYPE html>
+<html lang="en">
+    <head>
+        <meta charset="UTF-8">
+        <title>{{name}}</title>
+    </head>
+
+    <style>
+    * {
+        box-sizing: border-box;
+    }
+    .center-flex {
+        display: flex;
+        align-items: center;
+        justify-content: center;
+    }
+    .nav {
+        display: flex;
+        justify-content: space-between;
+    }
+    </style>
+    <body class="center-flex">
+{{body}}
+    </body>
+</html>
+"""
+
+
+def learning_module_template(body: bs4.Tag, name: str, prev: Optional[str], next: Optional[str]) -> str:
+    # Seems to be comments, ignore those.
+    for elem in body.select(".il-copg-mob-fullscreen-modal"):
+        elem.decompose()
+
+    nav_template = """
+        <div class="nav">
+            {{left}}
+            {{right}}
+        </div>
+    """
+    if prev and body.select_one(".ilc_page_lnav_LeftNavigation"):
+        text = body.select_one(".ilc_page_lnav_LeftNavigation").getText().strip()
+        left = f'<a href="{prev}">{text}</a>'
+    else:
+        left = "<span></span>"
+
+    if next and body.select_one(".ilc_page_rnav_RightNavigation"):
+        text = body.select_one(".ilc_page_rnav_RightNavigation").getText().strip()
+        right = f'<a href="{next}">{text}</a>'
+    else:
+        right = "<span></span>"
+
+    if top_nav := body.select_one(".ilc_page_tnav_TopNavigation"):
+        top_nav.replace_with(
+            soupify(nav_template.replace("{{left}}", left).replace("{{right}}", right).encode())
+        )
+
+    if bot_nav := body.select_one(".ilc_page_bnav_BottomNavigation"):
+        bot_nav.replace_with(soupify(nav_template.replace(
+            "{{left}}", left).replace("{{right}}", right).encode())
+        )
+
+    body = body.prettify()
+    return _learning_module_template.replace("{{body}}", body).replace("{{name}}", name)
+

 class Links(Enum):
    IGNORE = "ignore"
@ -102,24 +171,24 @@ class Links(Enum):
    INTERNET_SHORTCUT = "internet-shortcut"

    def template(self) -> Optional[str]:
-        if self == self.FANCY:
+        if self == Links.FANCY:
            return _link_template_fancy
-        elif self == self.PLAINTEXT:
+        elif self == Links.PLAINTEXT:
            return _link_template_plain
-        elif self == self.INTERNET_SHORTCUT:
+        elif self == Links.INTERNET_SHORTCUT:
            return _link_template_internet_shortcut
-        elif self == self.IGNORE:
+        elif self == Links.IGNORE:
            return None
        raise ValueError("Missing switch case")

    def extension(self) -> Optional[str]:
-        if self == self.FANCY:
+        if self == Links.FANCY:
            return ".html"
-        elif self == self.PLAINTEXT:
+        elif self == Links.PLAINTEXT:
            return ".txt"
-        elif self == self.INTERNET_SHORTCUT:
+        elif self == Links.INTERNET_SHORTCUT:
            return ".url"
-        elif self == self.IGNORE:
+        elif self == Links.IGNORE:
            return None
        raise ValueError("Missing switch case")

--- a/PFERD/crawl/ilias/ilias_html_cleaner.py
+++ b/PFERD/crawl/ilias/ilias_html_cleaner.py
@ -0,0 +1,103 @@
+from bs4 import BeautifulSoup, Comment, Tag
+
+_STYLE_TAG_CONTENT = """
+    .ilc_text_block_Information {
+      background-color: #f5f7fa;
+    }
+    div.ilc_text_block_Standard {
+      margin-bottom: 10px;
+      margin-top: 10px;
+    }
+    span.ilc_text_inline_Strong {
+      font-weight: bold;
+    }
+
+    .row-flex {
+      display: flex;
+    }
+    .row-flex-wrap {
+      flex-wrap: wrap;
+    }
+
+    .accordion-head {
+      background-color: #f5f7fa;
+      padding: 0.5rem 0;
+    }
+
+    h3 {
+      margin-top: 0.5rem;
+      margin-bottom: 1rem;
+    }
+
+    br.visible-break {
+      margin-bottom: 1rem;
+    }
+
+    article {
+      margin: 0.5rem 0;
+    }
+
+    body {
+      padding: 1em;
+      grid-template-columns: 1fr min(60rem, 90%) 1fr;
+      line-height: 1.2;
+    }
+"""
+
+_ARTICLE_WORTHY_CLASSES = [
+    "ilc_text_block_Information",
+    "ilc_section_Attention",
+    "ilc_section_Link",
+]
+
+
+def insert_base_markup(soup: BeautifulSoup) -> BeautifulSoup:
+    head = soup.new_tag("head")
+    soup.insert(0, head)
+
+    simplecss_link: Tag = soup.new_tag("link")
+    # <link rel="stylesheet" href="https://cdn.simplecss.org/simple.css">
+    simplecss_link["rel"] = "stylesheet"
+    simplecss_link["href"] = "https://cdn.simplecss.org/simple.css"
+    head.append(simplecss_link)
+
+    # Basic style tags for compat
+    style: Tag = soup.new_tag("style")
+    style.append(_STYLE_TAG_CONTENT)
+    head.append(style)
+
+    return soup
+
+
+def clean(soup: BeautifulSoup) -> BeautifulSoup:
+    for block in soup.find_all(class_=lambda x: x in _ARTICLE_WORTHY_CLASSES):
+        block.name = "article"
+
+    for block in soup.find_all("h3"):
+        block.name = "div"
+
+    for block in soup.find_all("h1"):
+        block.name = "h3"
+
+    for block in soup.find_all(class_="ilc_va_ihcap_VAccordIHeadCap"):
+        block.name = "h3"
+        block["class"] += ["accordion-head"]
+
+    for dummy in soup.select(".ilc_text_block_Standard.ilc_Paragraph"):
+        children = list(dummy.children)
+        if not children:
+            dummy.decompose()
+        if len(children) > 1:
+            continue
+        if isinstance(type(children[0]), Comment):
+            dummy.decompose()
+
+    # Delete video figures, as they can not be internalized anyway
+    for video in soup.select(".ilc_media_cont_MediaContainerHighlighted .ilPageVideo"):
+        if figure := video.find_parent("figure"):
+            figure.decompose()
+
+    for hrule_imposter in soup.find_all(class_="ilc_section_Separator"):
+        hrule_imposter.insert(0, soup.new_tag("hr"))
+
+    return soup
--- a/PFERD/crawl/ilias/ilias_web_crawler.py
+++ b/PFERD/crawl/ilias/ilias_web_crawler.py
--- a/PFERD/crawl/ilias/kit_ilias_html.py
+++ b/PFERD/crawl/ilias/kit_ilias_html.py
--- a/PFERD/crawl/ilias/kit_ilias_web_crawler.py
+++ b/PFERD/crawl/ilias/kit_ilias_web_crawler.py
@ -1,579 +1,37 @@
-import asyncio
-import re
-from pathlib import PurePath
-from typing import Any, Awaitable, Callable, Dict, List, Optional, Set, TypeVar, Union
+from typing import Dict, Literal

-import aiohttp
-from aiohttp import hdrs
-from bs4 import BeautifulSoup, Tag
-
-from ...auth import Authenticator, TfaAuthenticator
+from ...auth import Authenticator
 from ...config import Config
-from ...logging import ProgressBar, log
-from ...output_dir import FileSink, Redownload
-from ...utils import fmt_path, soupify, url_set_query_param
-from ..crawler import CrawlError, CrawlWarning, anoncritical
-from ..http_crawler import HttpCrawler, HttpCrawlerSection
-from .file_templates import Links
-from .kit_ilias_html import IliasElementType, IliasPage, IliasPageElement
+from .ilias_web_crawler import IliasWebCrawler, IliasWebCrawlerSection
+from .shibboleth_login import ShibbolethLogin

-TargetType = Union[str, int]
+_ILIAS_URL = "https://ilias.studium.kit.edu"


-class KitIliasWebCrawlerSection(HttpCrawlerSection):
-    def target(self) -> TargetType:
-        target = self.s.get("target")
-        if not target:
-            self.missing_value("target")
-
-        if re.fullmatch(r"\d+", target):
-            # Course id
-            return int(target)
-        if target == "desktop":
-            # Full personal desktop
-            return target
-        if target.startswith("https://ilias.studium.kit.edu"):
-            # ILIAS URL
-            return target
-
-        self.invalid_value("target", target, "Should be <course id | desktop | kit ilias URL>")
-
-    def tfa_auth(self, authenticators: Dict[str, Authenticator]) -> Optional[Authenticator]:
-        value: Optional[str] = self.s.get("tfa_auth")
-        if value is None:
-            return None
-        auth = authenticators.get(value)
-        if auth is None:
-            self.invalid_value("tfa_auth", value, "No such auth section exists")
-        return auth
-
-    def links(self) -> Links:
-        type_str: Optional[str] = self.s.get("links")
-
-        if type_str is None:
-            return Links.FANCY
-
-        try:
-            return Links.from_string(type_str)
-        except ValueError as e:
-            self.invalid_value("links", type_str, str(e).capitalize())
-
-    def link_redirect_delay(self) -> int:
-        return self.s.getint("link_redirect_delay", fallback=-1)
-
-    def videos(self) -> bool:
-        return self.s.getboolean("videos", fallback=False)
+class KitShibbolethBackgroundLoginSuccessful:
+    pass


-_DIRECTORY_PAGES: Set[IliasElementType] = set([
-    IliasElementType.EXERCISE,
-    IliasElementType.EXERCISE_FILES,
-    IliasElementType.FOLDER,
-    IliasElementType.MEETING,
-    IliasElementType.VIDEO_FOLDER,
-    IliasElementType.VIDEO_FOLDER_MAYBE_PAGINATED,
-])
+class KitIliasWebCrawlerSection(IliasWebCrawlerSection):
+    def base_url(self) -> str:
+        return _ILIAS_URL

-_VIDEO_ELEMENTS: Set[IliasElementType] = set([
-    IliasElementType.VIDEO,
-    IliasElementType.VIDEO_PLAYER,
-    IliasElementType.VIDEO_FOLDER,
-    IliasElementType.VIDEO_FOLDER_MAYBE_PAGINATED,
-])
-
-AWrapped = TypeVar("AWrapped", bound=Callable[..., Awaitable[None]])
+    def login(self) -> Literal["shibboleth"]:
+        return "shibboleth"


-def _iorepeat(attempts: int, name: str) -> Callable[[AWrapped], AWrapped]:
-    def decorator(f: AWrapped) -> AWrapped:
-        async def wrapper(*args: Any, **kwargs: Any) -> None:
-            last_exception: Optional[BaseException] = None
-            for round in range(attempts):
-                try:
-                    await f(*args, **kwargs)
-                    return
-                except aiohttp.ContentTypeError:  # invalid content type
-                    raise CrawlWarning("ILIAS returned an invalid content type")
-                except aiohttp.TooManyRedirects:
-                    raise CrawlWarning("Got stuck in a redirect loop")
-                except aiohttp.ClientPayloadError as e:  # encoding or not enough bytes
-                    last_exception = e
-                except aiohttp.ClientConnectionError as e:  # e.g. timeout, disconnect, resolve failed, etc.
-                    last_exception = e
-                except asyncio.exceptions.TimeoutError as e:  # explicit http timeouts in HttpCrawler
-                    last_exception = e
-                log.explain_topic(f"Retrying operation {name}. Retries left: {attempts - 1 - round}")
-
-            if last_exception:
-                message = f"Error in I/O Operation: {last_exception}"
-                raise CrawlWarning(message) from last_exception
-            raise CrawlError("Impossible return in ilias _iorepeat")
-
-        return wrapper  # type: ignore
-    return decorator
-
-
-def _wrap_io_in_warning(name: str) -> Callable[[AWrapped], AWrapped]:
-    """
-    Wraps any I/O exception in a CrawlWarning.
-    """
-    return _iorepeat(1, name)
-
-
-# Crawler control flow:
-#
-#     crawl_desktop -+
-#                    |
-#     crawl_course --+
-#                    |
-#     @_io_repeat    |        # retries internally (before the bar)
-#  +- crawl_url    <-+
-#  |
-#  |
-#  |  @_wrap_io_exception     # does not need to retry as children acquire bars
-#  +> crawl_ilias_element -+
-#  ^                       |
-#  |  @_io_repeat          |  # retries internally (before the bar)
-#  +- crawl_ilias_page <---+
-#  |                       |
-#  +> get_page             |  # Handles and retries authentication
-#                          |
-#     @_io_repeat          |  # retries internally (before the bar)
-#  +- download_link    <---+
-#  |                       |
-#  +> resolve_target       |  # Handles and retries authentication
-#                          |
-#     @_io_repeat          |  # retries internally (before the bar)
-#  +- download_video   <---+
-#  |                       |
-#  |  @_io_repeat          |  # retries internally (before the bar)
-#  +- download_file    <---+
-#  |
-#  +> stream_from_url         # Handles and retries authentication
-
-class KitIliasWebCrawler(HttpCrawler):
+class KitIliasWebCrawler(IliasWebCrawler):
    def __init__(
-            self,
-            name: str,
-            section: KitIliasWebCrawlerSection,
-            config: Config,
-            authenticators: Dict[str, Authenticator]
+        self,
+        name: str,
+        section: KitIliasWebCrawlerSection,
+        config: Config,
+        authenticators: Dict[str, Authenticator],
    ):
-        # Setting a main authenticator for cookie sharing
-        auth = section.auth(authenticators)
-        super().__init__(name, section, config, shared_auth=auth)
+        super().__init__(name, section, config, authenticators)

-        if section.tasks() > 1:
-            log.warn("""
-Please avoid using too many parallel requests as these are the KIT ILIAS
-instance's greatest bottleneck.
-            """.strip())
-
-        self._shibboleth_login = KitShibbolethLogin(
-            auth,
+        self._shibboleth_login = ShibbolethLogin(
+            _ILIAS_URL,
+            self._auth,
            section.tfa_auth(authenticators),
        )
-
-        self._base_url = "https://ilias.studium.kit.edu"
-
-        self._target = section.target()
-        self._link_file_redirect_delay = section.link_redirect_delay()
-        self._links = section.links()
-        self._videos = section.videos()
-
-    async def _run(self) -> None:
-        if isinstance(self._target, int):
-            log.explain_topic(f"Inferred crawl target: Course with id {self._target}")
-            await self._crawl_course(self._target)
-        elif self._target == "desktop":
-            log.explain_topic("Inferred crawl target: Personal desktop")
-            await self._crawl_desktop()
-        else:
-            log.explain_topic(f"Inferred crawl target: URL {self._target}")
-            await self._crawl_url(self._target)
-
-    async def _crawl_course(self, course_id: int) -> None:
-        # Start crawling at the given course
-        root_url = url_set_query_param(
-            self._base_url + "/goto.php", "target", f"crs_{course_id}"
-        )
-
-        await self._crawl_url(root_url, expected_id=course_id)
-
-    async def _crawl_desktop(self) -> None:
-        await self._crawl_url(self._base_url)
-
-    async def _crawl_url(self, url: str, expected_id: Optional[int] = None) -> None:
-        maybe_cl = await self.crawl(PurePath("."))
-        if not maybe_cl:
-            return
-        cl = maybe_cl  # Not mypy's fault, but explained here: https://github.com/python/mypy/issues/2608
-
-        elements: List[IliasPageElement] = []
-
-        @_iorepeat(3, "crawling url")
-        async def gather_elements() -> None:
-            elements.clear()
-            async with cl:
-                soup = await self._get_page(url)
-
-                if expected_id is not None:
-                    perma_link_element: Tag = soup.find(id="current_perma_link")
-                    if not perma_link_element or "crs_" not in perma_link_element.get("value"):
-                        raise CrawlError("Invalid course id? Didn't find anything looking like a course")
-
-                # Duplicated code, but the root page is special - we want to avoid fetching it twice!
-                log.explain_topic("Parsing root HTML page")
-                log.explain(f"URL: {url}")
-                page = IliasPage(soup, url, None)
-                elements.extend(page.get_child_elements())
-
-        # Fill up our task list with the found elements
-        await gather_elements()
-        tasks = [self._handle_ilias_element(PurePath("."), element) for element in elements]
-
-        # And execute them
-        await self.gather(tasks)
-
-    async def _handle_ilias_page(self, url: str, parent: IliasPageElement, path: PurePath) -> None:
-        maybe_cl = await self.crawl(path)
-        if not maybe_cl:
-            return
-        cl = maybe_cl  # Not mypy's fault, but explained here: https://github.com/python/mypy/issues/2608
-
-        elements: List[IliasPageElement] = []
-
-        @_iorepeat(3, "crawling folder")
-        async def gather_elements() -> None:
-            elements.clear()
-            async with cl:
-                next_stage_url: Optional[str] = url
-                current_parent = parent
-
-                while next_stage_url:
-                    soup = await self._get_page(next_stage_url)
-                    log.explain_topic(f"Parsing HTML page for {fmt_path(path)}")
-                    log.explain(f"URL: {next_stage_url}")
-                    page = IliasPage(soup, next_stage_url, current_parent)
-                    if next_element := page.get_next_stage_element():
-                        current_parent = next_element
-                        next_stage_url = next_element.url
-                    else:
-                        next_stage_url = None
-
-                elements.extend(page.get_child_elements())
-
-        # Fill up our task list with the found elements
-        await gather_elements()
-        tasks = [self._handle_ilias_element(cl.path, element) for element in elements]
-
-        # And execute them
-        await self.gather(tasks)
-
-    @anoncritical
-    # Shouldn't happen but we also really don't want to let I/O errors bubble up to anoncritical.
-    # If that happens we will be terminated as anoncritical doesn't tream them as non-critical.
-    @_wrap_io_in_warning("handling ilias element")
-    async def _handle_ilias_element(self, parent_path: PurePath, element: IliasPageElement) -> None:
-        element_path = PurePath(parent_path, element.name)
-
-        if element.type in _VIDEO_ELEMENTS:
-            log.explain_topic(f"Decision: Crawl video element {fmt_path(element_path)}")
-            if not self._videos:
-                log.explain("Video crawling is disabled")
-                log.explain("Answer: no")
-                return
-            else:
-                log.explain("Video crawling is enabled")
-                log.explain("Answer: yes")
-
-        if element.type == IliasElementType.FILE:
-            await self._download_file(element, element_path)
-        elif element.type == IliasElementType.FORUM:
-            log.explain_topic(f"Decision: Crawl {fmt_path(element_path)}")
-            log.explain("Forums are not supported")
-            log.explain("Answer: No")
-        elif element.type == IliasElementType.TEST:
-            log.explain_topic(f"Decision: Crawl {fmt_path(element_path)}")
-            log.explain("Tests contain no relevant files")
-            log.explain("Answer: No")
-        elif element.type == IliasElementType.LINK:
-            await self._download_link(element, element_path)
-        elif element.type == IliasElementType.VIDEO:
-            await self._download_file(element, element_path)
-        elif element.type == IliasElementType.VIDEO_PLAYER:
-            await self._download_video(element, element_path)
-        elif element.type in _DIRECTORY_PAGES:
-            await self._handle_ilias_page(element.url, element, element_path)
-        else:
-            # This will retry it a few times, failing everytime. It doesn't make any network
-            # requests, so that's fine.
-            raise CrawlWarning(f"Unknown element type: {element.type!r}")
-
-    async def _download_link(self, element: IliasPageElement, element_path: PurePath) -> None:
-        log.explain_topic(f"Decision: Crawl Link {fmt_path(element_path)}")
-        log.explain(f"Links type is {self._links}")
-
-        link_template_maybe = self._links.template()
-        link_extension = self._links.extension()
-        if not link_template_maybe or not link_extension:
-            log.explain("Answer: No")
-            return
-        else:
-            log.explain("Answer: Yes")
-        link_template = link_template_maybe
-        element_path = element_path.with_name(element_path.name + link_extension)
-
-        maybe_dl = await self.download(element_path, mtime=element.mtime)
-        if not maybe_dl:
-            return
-        dl = maybe_dl  # Not mypy's fault, but explained here: https://github.com/python/mypy/issues/2608
-
-        @_iorepeat(3, "resolving link")
-        async def impl() -> None:
-            async with dl as (bar, sink):
-                export_url = element.url.replace("cmd=calldirectlink", "cmd=exportHTML")
-                real_url = await self._resolve_link_target(export_url)
-
-                content = link_template
-                content = content.replace("{{link}}", real_url)
-                content = content.replace("{{name}}", element.name)
-                content = content.replace("{{description}}", str(element.description))
-                content = content.replace("{{redirect_delay}}", str(self._link_file_redirect_delay))
-                sink.file.write(content.encode("utf-8"))
-                sink.done()
-
-        await impl()
-
-    async def _resolve_link_target(self, export_url: str) -> str:
-        async with self.session.get(export_url, allow_redirects=False) as resp:
-            # No redirect means we were authenticated
-            if hdrs.LOCATION not in resp.headers:
-                return soupify(await resp.read()).select_one("a").get("href").strip()
-
-        self._authenticate()
-
-        async with self.session.get(export_url, allow_redirects=False) as resp:
-            # No redirect means we were authenticated
-            if hdrs.LOCATION not in resp.headers:
-                return soupify(await resp.read()).select_one("a").get("href").strip()
-
-        raise CrawlError("resolve_link_target failed even after authenticating")
-
-    async def _download_video(self, element: IliasPageElement, element_path: PurePath) -> None:
-        # Videos will NOT be redownloaded - their content doesn't really change and they are chunky
-        maybe_dl = await self.download(element_path, mtime=element.mtime, redownload=Redownload.NEVER)
-        if not maybe_dl:
-            return
-        dl = maybe_dl  # Not mypy's fault, but explained here: https://github.com/python/mypy/issues/2608
-
-        @_iorepeat(3, "downloading video")
-        async def impl() -> None:
-            assert dl  # The function is only reached when dl is not None
-            async with dl as (bar, sink):
-                page = IliasPage(await self._get_page(element.url), element.url, element)
-                real_element = page.get_child_elements()[0]
-
-                log.explain(f"Streaming video from real url {real_element.url}")
-
-                await self._stream_from_url(real_element.url, sink, bar, is_video=True)
-
-        await impl()
-
-    async def _download_file(self, element: IliasPageElement, element_path: PurePath) -> None:
-        maybe_dl = await self.download(element_path, mtime=element.mtime)
-        if not maybe_dl:
-            return
-        dl = maybe_dl  # Not mypy's fault, but explained here: https://github.com/python/mypy/issues/2608
-
-        @_iorepeat(3, "downloading file")
-        async def impl() -> None:
-            assert dl  # The function is only reached when dl is not None
-            async with dl as (bar, sink):
-                await self._stream_from_url(element.url, sink, bar, is_video=False)
-
-        await impl()
-
-    async def _stream_from_url(self, url: str, sink: FileSink, bar: ProgressBar, is_video: bool) -> None:
-        async def try_stream() -> bool:
-            async with self.session.get(url, allow_redirects=is_video) as resp:
-                if not is_video:
-                    # Redirect means we weren't authenticated
-                    if hdrs.LOCATION in resp.headers:
-                        return False
-                # we wanted a video but got HTML
-                if is_video and "html" in resp.content_type:
-                    return False
-
-                if resp.content_length:
-                    bar.set_total(resp.content_length)
-
-                async for data in resp.content.iter_chunked(1024):
-                    sink.file.write(data)
-                    bar.advance(len(data))
-
-                sink.done()
-            return True
-
-        auth_id = await self._current_auth_id()
-        if await try_stream():
-            return
-
-        await self.authenticate(auth_id)
-
-        if not await try_stream():
-            raise CrawlError("File streaming failed after authenticate()")
-
-    async def _get_page(self, url: str) -> BeautifulSoup:
-        auth_id = await self._current_auth_id()
-        async with self.session.get(url) as request:
-            soup = soupify(await request.read())
-            if self._is_logged_in(soup):
-                return soup
-
-        # We weren't authenticated, so try to do that
-        await self.authenticate(auth_id)
-
-        # Retry once after authenticating. If this fails, we will die.
-        async with self.session.get(url) as request:
-            soup = soupify(await request.read())
-            if self._is_logged_in(soup):
-                return soup
-        raise CrawlError("get_page failed even after authenticating")
-
-    # We repeat this as the login method in shibboleth doesn't handle I/O errors.
-    # Shibboleth is quite reliable as well, the repeat is likely not critical here.
-    @_iorepeat(3, "Login")
-    async def _authenticate(self) -> None:
-        await self._shibboleth_login.login(self.session)
-
-    @staticmethod
-    def _is_logged_in(soup: BeautifulSoup) -> bool:
-        # Normal ILIAS pages
-        userlog = soup.find("li", {"id": "userlog"})
-        if userlog is not None:
-            return True
-        # Video listing embeds do not have complete ILIAS html. Try to match them by
-        # their video listing table
-        video_table = soup.find(
-            recursive=True,
-            name="table",
-            attrs={"id": lambda x: x is not None and x.startswith("tbl_xoct")}
-        )
-        if video_table is not None:
-            return True
-        # The individual video player wrapper page has nothing of the above.
-        # Match it by its playerContainer.
-        if soup.select_one("#playerContainer") is not None:
-            return True
-        return False
-
-
-class KitShibbolethLogin:
-    """
-    Login via KIT's shibboleth system.
-    """
-
-    def __init__(self, authenticator: Authenticator, tfa_authenticator: Optional[Authenticator]) -> None:
-        self._auth = authenticator
-        self._tfa_auth = tfa_authenticator
-
-    async def login(self, sess: aiohttp.ClientSession) -> None:
-        """
-        Performs the ILIAS Shibboleth authentication dance and saves the login
-        cookies it receieves.
-
-        This function should only be called whenever it is detected that you're
-        not logged in. The cookies obtained should be good for a few minutes,
-        maybe even an hour or two.
-        """
-
-        # Equivalent: Click on "Mit KIT-Account anmelden" button in
-        # https://ilias.studium.kit.edu/login.php
-        url = "https://ilias.studium.kit.edu/Shibboleth.sso/Login"
-        data = {
-            "sendLogin": "1",
-            "idp_selection": "https://idp.scc.kit.edu/idp/shibboleth",
-            "target": "/shib_login.php",
-            "home_organization_selection": "Mit KIT-Account anmelden",
-        }
-        soup: BeautifulSoup = await _post(sess, url, data)
-
-        # Attempt to login using credentials, if necessary
-        while not self._login_successful(soup):
-            # Searching the form here so that this fails before asking for
-            # credentials rather than after asking.
-            form = soup.find("form", {"class": "full content", "method": "post"})
-            action = form["action"]
-
-            csrf_token = form.find("input", {"name": "csrf_token"})["value"]
-
-            # Equivalent: Enter credentials in
-            # https://idp.scc.kit.edu/idp/profile/SAML2/Redirect/SSO
-            url = "https://idp.scc.kit.edu" + action
-            username, password = await self._auth.credentials()
-            data = {
-                "_eventId_proceed": "",
-                "j_username": username,
-                "j_password": password,
-                "csrf_token": csrf_token
-            }
-            soup = await _post(sess, url, data)
-
-            if self._tfa_required(soup):
-                soup = await self._authenticate_tfa(sess, soup)
-
-            if not self._login_successful(soup):
-                self._auth.invalidate_credentials()
-
-        # Equivalent: Being redirected via JS automatically
-        # (or clicking "Continue" if you have JS disabled)
-        relay_state = soup.find("input", {"name": "RelayState"})
-        saml_response = soup.find("input", {"name": "SAMLResponse"})
-        url = "https://ilias.studium.kit.edu/Shibboleth.sso/SAML2/POST"
-        data = {  # using the info obtained in the while loop above
-            "RelayState": relay_state["value"],
-            "SAMLResponse": saml_response["value"],
-        }
-        await sess.post(url, data=data)
-
-    async def _authenticate_tfa(
-            self,
-            session: aiohttp.ClientSession,
-            soup: BeautifulSoup
-    ) -> BeautifulSoup:
-        if not self._tfa_auth:
-            self._tfa_auth = TfaAuthenticator("ilias-anon-tfa")
-
-        tfa_token = await self._tfa_auth.password()
-
-        # Searching the form here so that this fails before asking for
-        # credentials rather than after asking.
-        form = soup.find("form", {"method": "post"})
-        action = form["action"]
-        csrf_token = form.find("input", {"name": "csrf_token"})["value"]
-
-        # Equivalent: Enter token in
-        # https://idp.scc.kit.edu/idp/profile/SAML2/Redirect/SSO
-        url = "https://idp.scc.kit.edu" + action
-        data = {
-            "_eventId_proceed": "",
-            "j_tokenNumber": tfa_token,
-            "csrf_token": csrf_token
-        }
-        return await _post(session, url, data)
-
-    @staticmethod
-    def _login_successful(soup: BeautifulSoup) -> bool:
-        relay_state = soup.find("input", {"name": "RelayState"})
-        saml_response = soup.find("input", {"name": "SAMLResponse"})
-        return relay_state is not None and saml_response is not None
-
-    @staticmethod
-    def _tfa_required(soup: BeautifulSoup) -> bool:
-        return soup.find(id="j_tokenNumber") is not None
-
-
-async def _post(session: aiohttp.ClientSession, url: str, data: Any) -> BeautifulSoup:
-    async with session.post(url, data=data) as response:
-        return soupify(await response.read())
--- a/PFERD/crawl/ilias/shibboleth_login.py
+++ b/PFERD/crawl/ilias/shibboleth_login.py
@ -0,0 +1,128 @@
+from typing import Any, Optional
+
+import aiohttp
+import yarl
+from bs4 import BeautifulSoup
+
+from ...auth import Authenticator, TfaAuthenticator
+from ...logging import log
+from ...utils import soupify
+from ..crawler import CrawlError
+
+
+class ShibbolethLogin:
+    """
+    Login via shibboleth system.
+    """
+
+    def __init__(
+        self, ilias_url: str, authenticator: Authenticator, tfa_authenticator: Optional[Authenticator]
+    ) -> None:
+        self._ilias_url = ilias_url
+        self._auth = authenticator
+        self._tfa_auth = tfa_authenticator
+
+    async def login(self, sess: aiohttp.ClientSession) -> None:
+        """
+        Performs the ILIAS Shibboleth authentication dance and saves the login
+        cookies it receieves.
+
+        This function should only be called whenever it is detected that you're
+        not logged in. The cookies obtained should be good for a few minutes,
+        maybe even an hour or two.
+        """
+
+        # Equivalent: Click on "Mit KIT-Account anmelden" button in
+        # https://ilias.studium.kit.edu/login.php
+        url = f"{self._ilias_url}/shib_login.php"
+        async with sess.get(url) as response:
+            shib_url = response.url
+            if str(shib_url).startswith(self._ilias_url):
+                log.explain(
+                    "ILIAS recognized our shib token and logged us in in the background, returning"
+                )
+                return
+            soup: BeautifulSoup = soupify(await response.read())
+
+        # Attempt to login using credentials, if necessary
+        while not self._login_successful(soup):
+            # Searching the form here so that this fails before asking for
+            # credentials rather than after asking.
+            form = soup.find("form", {"method": "post"})
+            action = form["action"]
+
+            # Equivalent: Enter credentials in
+            # https://idp.scc.kit.edu/idp/profile/SAML2/Redirect/SSO
+            url = str(shib_url.origin()) + action
+            username, password = await self._auth.credentials()
+            data = {
+                "_eventId_proceed": "",
+                "j_username": username,
+                "j_password": password,
+            }
+            if csrf_token_input := form.find("input", {"name": "csrf_token"}):
+                data["csrf_token"] = csrf_token_input["value"]
+            soup = await _post(sess, url, data)
+
+            if soup.find(id="attributeRelease"):
+                raise CrawlError(
+                    "ILIAS Shibboleth entitlements changed! "
+                    "Please log in once in your browser and review them"
+                )
+
+            if self._tfa_required(soup):
+                soup = await self._authenticate_tfa(sess, soup, shib_url)
+
+            if not self._login_successful(soup):
+                self._auth.invalidate_credentials()
+
+        # Equivalent: Being redirected via JS automatically
+        # (or clicking "Continue" if you have JS disabled)
+        relay_state = soup.find("input", {"name": "RelayState"})
+        saml_response = soup.find("input", {"name": "SAMLResponse"})
+        url = form = soup.find("form", {"method": "post"})["action"]
+        data = {  # using the info obtained in the while loop above
+            "RelayState": relay_state["value"],
+            "SAMLResponse": saml_response["value"],
+        }
+        await sess.post(url, data=data)
+
+    async def _authenticate_tfa(
+        self, session: aiohttp.ClientSession, soup: BeautifulSoup, shib_url: yarl.URL
+    ) -> BeautifulSoup:
+        if not self._tfa_auth:
+            self._tfa_auth = TfaAuthenticator("ilias-anon-tfa")
+
+        tfa_token = await self._tfa_auth.password()
+
+        # Searching the form here so that this fails before asking for
+        # credentials rather than after asking.
+        form = soup.find("form", {"method": "post"})
+        action = form["action"]
+
+        # Equivalent: Enter token in
+        # https://idp.scc.kit.edu/idp/profile/SAML2/Redirect/SSO
+        url = str(shib_url.origin()) + action
+        username, password = await self._auth.credentials()
+        data = {
+            "_eventId_proceed": "",
+            "j_tokenNumber": tfa_token,
+        }
+        if csrf_token_input := form.find("input", {"name": "csrf_token"}):
+            data["csrf_token"] = csrf_token_input["value"]
+        return await _post(session, url, data)
+
+    @staticmethod
+    def _login_successful(soup: BeautifulSoup) -> bool:
+        relay_state = soup.find("input", {"name": "RelayState"})
+        saml_response = soup.find("input", {"name": "SAMLResponse"})
+        return relay_state is not None and saml_response is not None
+
+    @staticmethod
+    def _tfa_required(soup: BeautifulSoup) -> bool:
+        return soup.find(id="j_tokenNumber") is not None
+
+
+async def _post(session: aiohttp.ClientSession, url: str, data: Any) -> BeautifulSoup:
+    async with session.post(url, data=data) as response:
+        return soupify(await response.read())
--- a/PFERD/crawl/kit_ipd_crawler.py
+++ b/PFERD/crawl/kit_ipd_crawler.py
@ -0,0 +1,188 @@
+import os
+import re
+from dataclasses import dataclass
+from datetime import datetime
+from pathlib import PurePath
+from typing import Any, Awaitable, Generator, Iterable, List, Optional, Pattern, Tuple, Union
+from urllib.parse import urljoin
+
+from bs4 import BeautifulSoup, Tag
+
+from ..config import Config
+from ..logging import ProgressBar, log
+from ..output_dir import FileSink
+from ..utils import soupify
+from .crawler import CrawlError
+from .http_crawler import HttpCrawler, HttpCrawlerSection
+
+
+class KitIpdCrawlerSection(HttpCrawlerSection):
+    def target(self) -> str:
+        target = self.s.get("target")
+        if not target:
+            self.missing_value("target")
+
+        if not target.startswith("https://"):
+            self.invalid_value("target", target, "Should be a URL")
+
+        return target
+
+    def link_regex(self) -> Pattern[str]:
+        regex = self.s.get("link_regex", r"^.*?[^/]+\.(pdf|zip|c|cpp|java)$")
+        return re.compile(regex)
+
+
+@dataclass
+class KitIpdFile:
+    name: str
+    url: str
+
+    def explain(self) -> None:
+        log.explain(f"File {self.name!r} (href={self.url!r})")
+
+
+@dataclass
+class KitIpdFolder:
+    name: str
+    entries: List[Union[KitIpdFile, "KitIpdFolder"]]
+
+    def explain(self) -> None:
+        log.explain_topic(f"Folder {self.name!r}")
+        for entry in self.entries:
+            entry.explain()
+
+
+class KitIpdCrawler(HttpCrawler):
+
+    def __init__(
+            self,
+            name: str,
+            section: KitIpdCrawlerSection,
+            config: Config,
+    ):
+        super().__init__(name, section, config)
+        self._url = section.target()
+        self._file_regex = section.link_regex()
+
+    async def _run(self) -> None:
+        maybe_cl = await self.crawl(PurePath("."))
+        if not maybe_cl:
+            return
+
+        tasks: List[Awaitable[None]] = []
+
+        async with maybe_cl:
+            for item in await self._fetch_items():
+                item.explain()
+                if isinstance(item, KitIpdFolder):
+                    tasks.append(self._crawl_folder(PurePath("."), item))
+                else:
+                    log.explain_topic(f"Orphan file {item.name!r} (href={item.url!r})")
+                    log.explain("Attributing it to root folder")
+                    # do this here to at least be sequential and not parallel (rate limiting is hard, as the
+                    # crawl abstraction does not hold for these requests)
+                    etag, mtime = await self._request_resource_version(item.url)
+                    tasks.append(self._download_file(PurePath("."), item, etag, mtime))
+
+        await self.gather(tasks)
+
+    async def _crawl_folder(self, parent: PurePath, folder: KitIpdFolder) -> None:
+        path = parent / folder.name
+        if not await self.crawl(path):
+            return
+
+        tasks = []
+        for entry in folder.entries:
+            if isinstance(entry, KitIpdFolder):
+                tasks.append(self._crawl_folder(path, entry))
+            else:
+                # do this here to at least be sequential and not parallel (rate limiting is hard, as the crawl
+                # abstraction does not hold for these requests)
+                etag, mtime = await self._request_resource_version(entry.url)
+                tasks.append(self._download_file(path, entry, etag, mtime))
+
+        await self.gather(tasks)
+
+    async def _download_file(
+        self,
+        parent: PurePath,
+        file: KitIpdFile,
+        etag: Optional[str],
+        mtime: Optional[datetime]
+    ) -> None:
+        element_path = parent / file.name
+
+        prev_etag = self._get_previous_etag_from_report(element_path)
+        etag_differs = None if prev_etag is None else prev_etag != etag
+
+        maybe_dl = await self.download(element_path, etag_differs=etag_differs, mtime=mtime)
+        if not maybe_dl:
+            # keep storing the known file's etag
+            if prev_etag:
+                self._add_etag_to_report(element_path, prev_etag)
+            return
+
+        async with maybe_dl as (bar, sink):
+            await self._stream_from_url(file.url, element_path, sink, bar)
+
+    async def _fetch_items(self) -> Iterable[Union[KitIpdFile, KitIpdFolder]]:
+        page, url = await self.get_page()
+        elements: List[Tag] = self._find_file_links(page)
+
+        # do not add unnecessary nesting for a single <h1> heading
+        drop_h1: bool = len(page.find_all(name="h1")) <= 1
+
+        folder_tree: KitIpdFolder = KitIpdFolder(".", [])
+        for element in elements:
+            parent = HttpCrawler.get_folder_structure_from_heading_hierarchy(element, drop_h1)
+            file = self._extract_file(element, url)
+
+            current_folder: KitIpdFolder = folder_tree
+            for folder_name in parent.parts:
+                # helps the type checker to verify that current_folder is indeed a folder
+                def subfolders() -> Generator[KitIpdFolder, Any, None]:
+                    return (entry for entry in current_folder.entries if isinstance(entry, KitIpdFolder))
+
+                if not any(entry.name == folder_name for entry in subfolders()):
+                    current_folder.entries.append(KitIpdFolder(folder_name, []))
+                current_folder = next(entry for entry in subfolders() if entry.name == folder_name)
+
+            current_folder.entries.append(file)
+
+        return folder_tree.entries
+
+    def _extract_file(self, link: Tag, url: str) -> KitIpdFile:
+        url = self._abs_url_from_link(url, link)
+        name = os.path.basename(url)
+        return KitIpdFile(name, url)
+
+    def _find_file_links(self, tag: Union[Tag, BeautifulSoup]) -> List[Tag]:
+        return tag.findAll(name="a", attrs={"href": self._file_regex})
+
+    def _abs_url_from_link(self, url: str, link_tag: Tag) -> str:
+        return urljoin(url, link_tag.get("href"))
+
+    async def _stream_from_url(self, url: str, path: PurePath, sink: FileSink, bar: ProgressBar) -> None:
+        async with self.session.get(url, allow_redirects=False) as resp:
+            if resp.status == 403:
+                raise CrawlError("Received a 403. Are you within the KIT network/VPN?")
+            if resp.content_length:
+                bar.set_total(resp.content_length)
+
+            async for data in resp.content.iter_chunked(1024):
+                sink.file.write(data)
+                bar.advance(len(data))
+
+            sink.done()
+
+            self._add_etag_to_report(path, resp.headers.get("ETag"))
+
+    async def get_page(self) -> Tuple[BeautifulSoup, str]:
+        async with self.session.get(self._url) as request:
+            # The web page for Algorithmen für Routenplanung contains some
+            # weird comments that beautifulsoup doesn't parse correctly. This
+            # hack enables those pages to be crawled, and should hopefully not
+            # cause issues on other pages.
+            content = (await request.read()).decode("utf-8")
+            content = re.sub(r"<!--.*?-->", "", content)
+            return soupify(content.encode("utf-8")), str(request.url)
--- a/PFERD/deduplicator.py
+++ b/PFERD/deduplicator.py
@ -14,7 +14,7 @@ def name_variants(path: PurePath) -> Iterator[PurePath]:


 class Deduplicator:
-    FORBIDDEN_CHARS = '<>:"/\\|?*'
+    FORBIDDEN_CHARS = '<>:"/\\|?*' + "".join([chr(i) for i in range(0, 32)])
    FORBIDDEN_NAMES = {
        "CON", "PRN", "AUX", "NUL",
        "COM1", "COM2", "COM3", "COM4", "COM5", "COM6", "COM7", "COM8", "COM9",
@ -56,6 +56,12 @@ class Deduplicator:
            log.explain(f"Changed path to {fmt_path(new_path)} for windows compatibility")
        return new_path

+    def fixup_path(self, path: PurePath) -> PurePath:
+        """Fixes up the path for windows, if enabled. Returns the path unchanged otherwise."""
+        if self._windows_paths:
+            return self._fixup_for_windows(path)
+        return path
+
    def mark(self, path: PurePath) -> PurePath:
        if self._windows_paths:
            path = self._fixup_for_windows(path)
--- a/PFERD/logging.py
+++ b/PFERD/logging.py
@ -5,7 +5,7 @@ from contextlib import asynccontextmanager, contextmanager
 # TODO In Python 3.9 and above, ContextManager is deprecated
 from typing import AsyncIterator, ContextManager, Iterator, List, Optional

-from rich.console import Console, RenderGroup
+from rich.console import Console, Group
 from rich.live import Live
 from rich.markup import escape
 from rich.panel import Panel
@ -59,6 +59,7 @@ class Log:
        # Whether different parts of the output are enabled or disabled
        self.output_explain = False
        self.output_status = True
+        self.output_not_deleted = True
        self.output_report = True

    def _update_live(self) -> None:
@ -68,7 +69,7 @@ class Log:
        if self._download_progress.task_ids:
            elements.append(self._download_progress)

-        group = RenderGroup(*elements)  # type: ignore
+        group = Group(*elements)
        self._live.update(group)

    @contextmanager
@ -197,7 +198,7 @@ directly or as a GitHub issue: https://github.com/Garmelon/PFERD/issues/new
        if self.output_explain:
            self.print(f"  {escape(text)}")

-    def status(self, style: str, action: str, text: str) -> None:
+    def status(self, style: str, action: str, text: str, suffix: str = "") -> None:
        """
        Print a status update while crawling. Allows markup in the "style"
        argument which will be applied to the "action" string.
@ -205,7 +206,18 @@ directly or as a GitHub issue: https://github.com/Garmelon/PFERD/issues/new

        if self.output_status:
            action = escape(f"{action:<{self.STATUS_WIDTH}}")
-            self.print(f"{style}{action}[/] {escape(text)}")
+            self.print(f"{style}{action}[/] {escape(text)} {suffix}")
+
+    def not_deleted(self, style: str, action: str, text: str, suffix: str = "") -> None:
+        """
+        Print a message for a local only file that wasn't
+        deleted while crawling. Allows markup in the "style"
+        argument which will be applied to the "action" string.
+        """
+
+        if self.output_status and self.output_not_deleted:
+            action = escape(f"{action:<{self.STATUS_WIDTH}}")
+            self.print(f"{style}{action}[/] {escape(text)} {suffix}")

    def report(self, text: str) -> None:
        """
@ -215,6 +227,14 @@ directly or as a GitHub issue: https://github.com/Garmelon/PFERD/issues/new
        if self.output_report:
            self.print(text)

+    def report_not_deleted(self, text: str) -> None:
+        """
+        Print a report for a local only file that wasn't deleted after crawling. Allows markup.
+        """
+
+        if self.output_report and self.output_not_deleted:
+            self.print(text)
+
    @contextmanager
    def _bar(
            self,
--- a/PFERD/output_dir.py
+++ b/PFERD/output_dir.py
@ -44,6 +44,7 @@ class OnConflict(Enum):
    LOCAL_FIRST = "local-first"
    REMOTE_FIRST = "remote-first"
    NO_DELETE = "no-delete"
+    NO_DELETE_PROMPT_OVERWRITE = "no-delete-prompt-overwrite"

    @staticmethod
    def from_string(string: str) -> "OnConflict":
@ -51,11 +52,12 @@ class OnConflict(Enum):
            return OnConflict(string)
        except ValueError:
            raise ValueError("must be one of 'prompt', 'local-first',"
-                             " 'remote-first', 'no-delete'")
+                             " 'remote-first', 'no-delete', 'no-delete-prompt-overwrite'")


@dataclass
 class Heuristics:
+    etag_differs: Optional[bool]
    mtime: Optional[datetime]


@ -231,7 +233,18 @@ class OutputDirectory:
        stat = local_path.stat()

        remote_newer = None
-        if mtime := heuristics.mtime:
+
+        # ETag should be a more reliable indicator than mtime, so we check it first
+        if heuristics.etag_differs is not None:
+            remote_newer = heuristics.etag_differs
+            if remote_newer:
+                log.explain("Remote file's entity tag differs")
+            else:
+                log.explain("Remote file's entity tag is the same")
+
+        # Python on Windows crashes when faced with timestamps around the unix epoch
+        if remote_newer is None and heuristics.mtime and (os.name != "nt" or heuristics.mtime.year > 1970):
+            mtime = heuristics.mtime
            remote_newer = mtime.timestamp() > stat.st_mtime
            if remote_newer:
                log.explain("Remote file seems to be newer")
@ -261,7 +274,7 @@ class OutputDirectory:
            on_conflict: OnConflict,
            path: PurePath,
    ) -> bool:
-        if on_conflict == OnConflict.PROMPT:
+        if on_conflict in {OnConflict.PROMPT, OnConflict.NO_DELETE_PROMPT_OVERWRITE}:
            async with log.exclusive_output():
                prompt = f"Replace {fmt_path(path)} with remote file?"
                return await prompt_yes_no(prompt, default=False)
@ -280,7 +293,7 @@ class OutputDirectory:
            on_conflict: OnConflict,
            path: PurePath,
    ) -> bool:
-        if on_conflict == OnConflict.PROMPT:
+        if on_conflict in {OnConflict.PROMPT, OnConflict.NO_DELETE_PROMPT_OVERWRITE}:
            async with log.exclusive_output():
                prompt = f"Recursively delete {fmt_path(path)} and replace with remote file?"
                return await prompt_yes_no(prompt, default=False)
@ -300,7 +313,7 @@ class OutputDirectory:
            path: PurePath,
            parent: PurePath,
    ) -> bool:
-        if on_conflict == OnConflict.PROMPT:
+        if on_conflict in {OnConflict.PROMPT, OnConflict.NO_DELETE_PROMPT_OVERWRITE}:
            async with log.exclusive_output():
                prompt = f"Delete {fmt_path(parent)} so remote file {fmt_path(path)} can be downloaded?"
                return await prompt_yes_no(prompt, default=False)
@ -327,7 +340,7 @@ class OutputDirectory:
            return False
        elif on_conflict == OnConflict.REMOTE_FIRST:
            return True
-        elif on_conflict == OnConflict.NO_DELETE:
+        elif on_conflict in {OnConflict.NO_DELETE, OnConflict.NO_DELETE_PROMPT_OVERWRITE}:
            return False

        # This should never be reached
@ -362,6 +375,8 @@ class OutputDirectory:
            self,
            remote_path: PurePath,
            path: PurePath,
+            *,
+            etag_differs: Optional[bool] = None,
            mtime: Optional[datetime] = None,
            redownload: Optional[Redownload] = None,
            on_conflict: Optional[OnConflict] = None,
@ -371,7 +386,7 @@ class OutputDirectory:
        MarkConflictError.
        """

-        heuristics = Heuristics(mtime)
+        heuristics = Heuristics(etag_differs, mtime)
        redownload = self._redownload if redownload is None else redownload
        on_conflict = self._on_conflict if on_conflict is None else on_conflict
        local_path = self.resolve(path)
@ -492,7 +507,7 @@ class OutputDirectory:
            except OSError:
                pass
        else:
-            log.status("[bold bright_magenta]", "Not deleted", fmt_path(pure))
+            log.not_deleted("[bold bright_magenta]", "Not deleted", fmt_path(pure))
            self._report.not_delete_file(pure)

    def load_prev_report(self) -> None:
@ -500,7 +515,7 @@ class OutputDirectory:
        try:
            self._prev_report = Report.load(self._report_path)
            log.explain("Loaded report successfully")
-        except (OSError, json.JSONDecodeError, ReportLoadError) as e:
+        except (OSError, UnicodeDecodeError, json.JSONDecodeError, ReportLoadError) as e:
            log.explain("Failed to load report")
            log.explain(str(e))

--- a/PFERD/pferd.py
+++ b/PFERD/pferd.py
@ -1,5 +1,6 @@
-from pathlib import Path
+from pathlib import Path, PurePath
 from typing import Dict, List, Optional
+from urllib.parse import quote

 from rich.markup import escape

@ -15,13 +16,13 @@ class PferdLoadError(Exception):


 class Pferd:
-    def __init__(self, config: Config, cli_crawlers: Optional[List[str]]):
+    def __init__(self, config: Config, cli_crawlers: Optional[List[str]], cli_skips: Optional[List[str]]):
        """
        May throw PferdLoadError.
        """

        self._config = config
-        self._crawlers_to_run = self._find_crawlers_to_run(config, cli_crawlers)
+        self._crawlers_to_run = self._find_crawlers_to_run(config, cli_crawlers, cli_skips)

        self._authenticators: Dict[str, Authenticator] = {}
        self._crawlers: Dict[str, Crawler] = {}
@ -65,16 +66,30 @@ class Pferd:

        return crawlers_to_run

-    def _find_crawlers_to_run(self, config: Config, cli_crawlers: Optional[List[str]]) -> List[str]:
+    def _find_crawlers_to_run(
+            self,
+            config: Config,
+            cli_crawlers: Optional[List[str]],
+            cli_skips: Optional[List[str]],
+    ) -> List[str]:
        log.explain_topic("Deciding which crawlers to run")

+        crawlers: List[str]
        if cli_crawlers is None:
            log.explain("No crawlers specified on CLI")
            log.explain("Running crawlers specified in config")
-            return self._find_config_crawlers(config)
+            crawlers = self._find_config_crawlers(config)
        else:
            log.explain("Crawlers specified on CLI")
-            return self._find_cli_crawlers(config, cli_crawlers)
+            crawlers = self._find_cli_crawlers(config, cli_crawlers)
+
+        skips = {f"crawl:{name}" for name in cli_skips} if cli_skips else set()
+        for crawler in crawlers:
+            if crawler in skips:
+                log.explain(f"Skipping crawler {crawler!r}")
+        crawlers = [crawler for crawler in crawlers if crawler not in skips]
+
+        return crawlers

    def _load_authenticators(self) -> None:
        for name, section in self._config.auth_sections():
@ -154,19 +169,32 @@ class Pferd:
            log.report("")
            log.report(f"[bold bright_cyan]Report[/] for {escape(name)}")

+            def fmt_path_link(relative_path: PurePath) -> str:
+                # We need to URL-encode the path because it might contain spaces or special characters
+                link = f"file://{quote(str(crawler.output_dir.resolve(relative_path).absolute()))}"
+                return f"[link={link}]{fmt_path(relative_path)}[/link]"
+
            something_changed = False
            for path in sorted(crawler.report.added_files):
                something_changed = True
-                log.report(f"  [bold bright_green]Added[/] {fmt_path(path)}")
+                log.report(f"  [bold bright_green]Added[/] {fmt_path_link(path)}")
            for path in sorted(crawler.report.changed_files):
                something_changed = True
-                log.report(f"  [bold bright_yellow]Changed[/] {fmt_path(path)}")
+                log.report(f"  [bold bright_yellow]Changed[/] {fmt_path_link(path)}")
            for path in sorted(crawler.report.deleted_files):
                something_changed = True
                log.report(f"  [bold bright_magenta]Deleted[/] {fmt_path(path)}")
            for path in sorted(crawler.report.not_deleted_files):
                something_changed = True
-                log.report(f"  [bold bright_magenta]Not deleted[/] {fmt_path(path)}")
+                log.report_not_deleted(f"  [bold bright_magenta]Not deleted[/] {fmt_path_link(path)}")
+
+            for warning in crawler.report.encountered_warnings:
+                something_changed = True
+                log.report(f"  [bold bright_red]Warning[/] {warning}")
+
+            for error in crawler.report.encountered_errors:
+                something_changed = True
+                log.report(f"  [bold bright_red]Error[/] {error}")

            if not something_changed:
                log.report("  Nothing changed")
--- a/PFERD/report.py
+++ b/PFERD/report.py
@ -1,6 +1,6 @@
 import json
 from pathlib import Path, PurePath
-from typing import Any, Dict, List, Set
+from typing import Any, Dict, List, Optional, Set


 class ReportLoadError(Exception):
@ -68,6 +68,13 @@ class Report:
        # Files that should have been deleted by the cleanup but weren't
        self.not_deleted_files: Set[PurePath] = set()

+        # Custom crawler-specific data
+        self.custom: Dict[str, Any] = dict()
+
+        # Encountered errors and warnings
+        self.encountered_warnings: List[str] = []
+        self.encountered_errors: List[str] = []
+
    @staticmethod
    def _get_list_of_strs(data: Dict[str, Any], key: str) -> List[str]:
        result: Any = data.get(key, [])
@ -81,13 +88,22 @@ class Report:

        return result

+    @staticmethod
+    def _get_str_dictionary(data: Dict[str, Any], key: str) -> Dict[str, Any]:
+        result: Dict[str, Any] = data.get(key, {})
+
+        if not isinstance(result, dict):
+            raise ReportLoadError(f"Incorrect format: {key!r} is not a dictionary")
+
+        return result
+
    @classmethod
    def load(cls, path: Path) -> "Report":
        """
-        May raise OSError, JsonDecodeError, ReportLoadError.
+        May raise OSError, UnicodeDecodeError, JsonDecodeError, ReportLoadError.
        """

-        with open(path) as f:
+        with open(path, encoding="utf-8") as f:
            data = json.load(f)

        if not isinstance(data, dict):
@ -108,6 +124,9 @@ class Report:
            self.delete_file(PurePath(elem))
        for elem in self._get_list_of_strs(data, "not_deleted"):
            self.not_delete_file(PurePath(elem))
+        self.custom = self._get_str_dictionary(data, "custom")
+        self.encountered_errors = self._get_list_of_strs(data, "encountered_errors")
+        self.encountered_warnings = self._get_list_of_strs(data, "encountered_warnings")

        return self

@ -124,9 +143,12 @@ class Report:
            "changed": [str(path) for path in sorted(self.changed_files)],
            "deleted": [str(path) for path in sorted(self.deleted_files)],
            "not_deleted": [str(path) for path in sorted(self.not_deleted_files)],
+            "custom": self.custom,
+            "encountered_warnings": self.encountered_warnings,
+            "encountered_errors": self.encountered_errors,
        }

-        with open(path, "w") as f:
+        with open(path, "w", encoding="utf-8") as f:
            json.dump(data, f, indent=2, sort_keys=True)
            f.write("\n")  # json.dump doesn't do this

@ -190,3 +212,27 @@ class Report:
        """

        self.not_deleted_files.add(path)
+
+    def add_custom_value(self, key: str, value: Any) -> None:
+        """
+        Adds a custom value under the passed key, overwriting any existing
+        """
+        self.custom[key] = value
+
+    def get_custom_value(self, key: str) -> Optional[Any]:
+        """
+        Retrieves a custom value for the given key.
+        """
+        return self.custom.get(key)
+
+    def add_error(self, error: str) -> None:
+        """
+        Adds an error to this report's error list.
+        """
+        self.encountered_errors.append(error)
+
+    def add_warning(self, warning: str) -> None:
+        """
+        Adds a warning to this report's warning list.
+        """
+        self.encountered_warnings.append(warning)
--- a/PFERD/transformer.py
+++ b/PFERD/transformer.py
@ -41,9 +41,11 @@ TransformResult = Optional[Union[Transformed, Ignored]]
@dataclass
 class Rule:
    left: str
+    left_index: int
    name: str
    head: ArrowHead
    right: RightSide
+    right_index: int

    def right_result(self, path: PurePath) -> Union[str, Transformed, Ignored]:
        if isinstance(self.right, str):
@ -108,6 +110,10 @@ class ExactReTf(Transformation):
            except ValueError:
                pass

+        named_groups: Dict[str, str] = match.groupdict()
+        for name, capture in named_groups.items():
+            locals_dir[name] = capture
+
        result = eval(f"f{right!r}", {}, locals_dir)
        return Transformed(PurePath(result))

@ -345,6 +351,7 @@ def parse_eol(line: Line) -> None:

 def parse_rule(line: Line) -> Rule:
    parse_zero_or_more_spaces(line)
+    left_index = line.index
    left = parse_left(line)

    parse_one_or_more_spaces(line)
@ -354,19 +361,19 @@ def parse_rule(line: Line) -> Rule:
    line.expect("-")
    head = parse_arrow_head(line)

-    index = line.index
+    right_index = line.index
    right: RightSide
    try:
        parse_zero_or_more_spaces(line)
        parse_eol(line)
        right = Empty()
    except RuleParseError:
-        line.index = index
+        line.index = right_index
        parse_one_or_more_spaces(line)
        right = parse_right(line)
        parse_eol(line)

-    return Rule(left, name, head, right)
+    return Rule(left, left_index, name, head, right, right_index)


 def parse_transformation(line: Line) -> Transformation:
@ -377,6 +384,9 @@ def parse_transformation(line: Line) -> Transformation:
    elif rule.name == "exact":
        return ExactTf(rule)
    elif rule.name == "name":
+        if len(PurePath(rule.left).parts) > 1:
+            line.index = rule.left_index
+            raise RuleParseError(line, "Expected name, not multiple segments")
        return RenamingPartsTf(ExactTf(rule))
    elif rule.name == "re":
        return RenamingParentsTf(ExactReTf(rule))
--- a/PFERD/version.py
+++ b/PFERD/version.py
@ -1,2 +1,2 @@
 NAME = "PFERD"
-VERSION = "3.1.0"
+VERSION = "3.7.0"
--- a/README.md
+++ b/README.md
@ -17,7 +17,7 @@ Binaries for Linux, Windows and Mac can be downloaded directly from the

 ### With pip

-Ensure you have at least Python 3.8 installed. Run the following command to
+Ensure you have at least Python 3.9 installed. Run the following command to
 install PFERD or upgrade it to the latest version:

 ```
@ -26,6 +26,17 @@ $ pip install --upgrade git+https://github.com/Garmelon/PFERD@latest

 The use of [venv](https://docs.python.org/3/library/venv.html) is recommended.

+### With package managers
+
+Unofficial packages are available for:
+- [AUR](https://aur.archlinux.org/packages/pferd)
+- [brew](https://formulae.brew.sh/formula/pferd)
+- [conda-forge](https://github.com/conda-forge/pferd-feedstock)
+- [nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/misc/pferd/default.nix)
+- [PyPi](https://pypi.org/project/pferd)
+
+See also PFERD's [repology page](https://repology.org/project/pferd/versions).
+
 ## Basic usage

 PFERD can be run directly from the command line with no config file. Run `pferd
@ -45,6 +56,17 @@ Also, you can download most ILIAS pages directly like this:
 $ pferd kit-ilias-web <url> <output_directory>
 ```

+PFERD supports other ILIAS instances as well, using the `ilias-web` crawler (see
+the [config section on `ilias-web`](CONFIG.md#the-ilias-web-crawler) for more
+detail on the `base-url` and `client-id` parameters):
+
+```
+$ pferd ilias-web \
+    --base-url https://ilias.my-university.example \
+    --client-id My_University desktop \
+    <output_directory>
+```
+
 However, the CLI only lets you download a single thing at a time, and the
 resulting command can grow long quite quickly. Because of this, PFERD can also
 be used with a config file.
--- a/flake.lock
+++ b/flake.lock
@ -0,0 +1,27 @@
+{
+  "nodes": {
+    "nixpkgs": {
+      "locked": {
+        "lastModified": 1708979614,
+        "narHash": "sha256-FWLWmYojIg6TeqxSnHkKpHu5SGnFP5um1uUjH+wRV6g=",
+        "owner": "NixOS",
+        "repo": "nixpkgs",
+        "rev": "b7ee09cf5614b02d289cd86fcfa6f24d4e078c2a",
+        "type": "github"
+      },
+      "original": {
+        "owner": "NixOS",
+        "ref": "nixos-23.11",
+        "repo": "nixpkgs",
+        "type": "github"
+      }
+    },
+    "root": {
+      "inputs": {
+        "nixpkgs": "nixpkgs"
+      }
+    }
+  },
+  "root": "root",
+  "version": 7
+}
--- a/flake.nix
+++ b/flake.nix
@ -0,0 +1,41 @@
+{
+  description = "Tool for downloading course-related files from ILIAS";
+
+  inputs = {
+    nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.11";
+  };
+
+  outputs = { self, nixpkgs }:
+    let
+      # Helper function to generate an attrset '{ x86_64-linux = f "x86_64-linux"; ... }'.
+      forAllSystems = nixpkgs.lib.genAttrs nixpkgs.lib.systems.flakeExposed;
+    in
+    {
+      packages = forAllSystems (system:
+        let pkgs = import nixpkgs { inherit system; };
+        in
+        rec {
+          default = pkgs.python3Packages.buildPythonApplication rec {
+            pname = "pferd";
+            # Performing black magic
+            # Don't worry, I sacrificed enough goats for the next few years
+            version = (pkgs.lib.importTOML ./PFERD/version.py).VERSION;
+            format = "pyproject";
+
+            src = ./.;
+
+            nativeBuildInputs = with pkgs.python3Packages; [
+              setuptools
+            ];
+
+            propagatedBuildInputs = with pkgs.python3Packages; [
+              aiohttp
+              beautifulsoup4
+              rich
+              keyring
+              certifi
+            ];
+          };
+        });
+    };
+}
--- a/mypy.ini
+++ b/mypy.ini
@ -1,11 +0,0 @@
-[mypy]
-disallow_any_generics = True
-disallow_untyped_defs = True
-disallow_incomplete_defs = True
-no_implicit_optional = True
-warn_unused_ignores = True
-warn_unreachable = True
-show_error_context = True
-
-[mypy-rich.*,bs4,keyring]
-ignore_missing_imports = True
--- a/pyproject.toml
+++ b/pyproject.toml
@ -1,3 +1,42 @@
 [build-system]
 requires = ["setuptools", "wheel"]
 build-backend = "setuptools.build_meta"
+
+[project]
+name = "PFERD"
+dependencies = [
+  "aiohttp>=3.8.1",
+  "beautifulsoup4>=4.10.0",
+  "rich>=11.0.0",
+  "keyring>=23.5.0",
+  "certifi>=2021.10.8"
+]
+dynamic = ["version"]
+requires-python = ">=3.9"
+
+[project.scripts]
+pferd = "PFERD.__main__:main"
+
+[tool.setuptools.dynamic]
+version = {attr = "PFERD.version.VERSION"}
+
+[tool.flake8]
+max-line-length = 110
+
+[tool.isort]
+line_length = 110
+
+[tool.autopep8]
+max_line_length = 110
+in-place = true
+recursive = true
+
+[tool.mypy]
+disallow_any_generics = true
+disallow_untyped_defs = true
+disallow_incomplete_defs = true
+no_implicit_optional = true
+warn_unused_ignores = true
+warn_unreachable = true
+show_error_context = true
+ignore_missing_imports = true
--- a/scripts/bump-version
+++ b/scripts/bump-version
@ -1,8 +1,8 @@
 #!/usr/bin/env python3

 import argparse
-import time
 import re
+import time
 from subprocess import run


--- a/scripts/check
+++ b/scripts/check
@ -2,5 +2,5 @@

 set -e

-mypy PFERD
+mypy .
 flake8 PFERD
--- a/scripts/format
+++ b/scripts/format
@ -2,5 +2,5 @@

 set -e

-autopep8 --recursive --in-place PFERD
-isort PFERD
+autopep8 .
+isort .
--- a/scripts/setup
+++ b/scripts/setup
@ -13,5 +13,5 @@ pip install --upgrade setuptools
 pip install --editable .

 # Installing tools and type hints
-pip install --upgrade mypy flake8 autopep8 isort pyinstaller
+pip install --upgrade mypy flake8 flake8-pyproject autopep8 isort pyinstaller
 pip install --upgrade types-chardet types-certifi
--- a/setup.cfg
+++ b/setup.cfg
@ -1,23 +0,0 @@
-[metadata]
-name = PFERD
-version = attr: PFERD.version.VERSION
-
-[options]
-packages = find:
-python_requires = >=3.8
-install_requires =
-  aiohttp>=3.7.4.post0
-  beautifulsoup4>=4.9.3
-  rich>=10.1.0
-  keyring>=23.0.1
-  certifi>=2020.12.5
-
-[options.entry_points]
-console_scripts =
-  pferd = PFERD.__main__:main
-
-[flake8]
-max_line_length = 110
-
-[isort]
-line_length = 110
Author	SHA1	Message	Date
I-Al-Istannen	287173b0b1	Bump version to 3.7.0	2024-11-13 20:38:27 +01:00
I-Al-Istannen	712217e959	Handle groups in cards	2024-11-11 12:53:08 +01:00
I-Al-Istannen	6dda4c55a8	Add doctype header to forum threads This should fix mimetype detection on most systems and is more relevant now that the report is clickable	2024-11-05 18:36:21 +01:00
PinieP	596b6a7688	Add support for non-KIT shibboleth login (#98 ) Co-authored-by: Mr-Pine <git@mr-pine.de> Co-authored-by: I-Al-Istannen <I-Al-Istannen@users.noreply.github.com>	2024-11-05 18:30:34 +01:00
Tim	5983200247	Treat headings as folders in kit-ipd crawler (#99 )	2024-11-04 23:53:48 +01:00
Tim	26e802d88b	Add clickable links to file names in the printed report (#100 ) Co-authored-by: I-Al-Istannen <i-al-istannen@users.noreply.github.com>	2024-11-04 00:32:32 +01:00
I-Al-Istannen	f5c4e82816	Delay ilias loop detection after transform This allows users to filter out duplicated elements and suppress the warning.	2024-11-02 22:46:51 +01:00
I-Al-Istannen	f5273f7ca0	Collapse ilias url crawling into normal page crawling	2024-11-02 22:46:51 +01:00
I-Al-Istannen	fa71a9f44f	Add support for mob videos in page descriptions	2024-10-28 20:35:30 +01:00
I-Al-Istannen	81d6ff53c4	Respect row flex in descriptions	2024-10-28 19:41:03 +01:00
I-Al-Istannen	d7a2b6e019	Delete videos from course descriptions	2024-10-28 19:41:03 +01:00
I-Al-Istannen	71c65e89d1	Internalize images in course descriptions	2024-10-28 19:41:03 +01:00
I-Al-Istannen	c1046498e7	Fix download of links without a target URL They are now downloaded as links to the empty url.	2024-10-28 19:41:03 +01:00
I-Al-Istannen	8fbd1978af	Fix crawling of nested courses	2024-10-28 18:52:27 +01:00
Tim	739dd95850	Use Last-Modified and ETag headers to determine KIT-IPD file versions (#95 ) Co-authored-by: I-Al-Istannen <i-al-istannen@users.noreply.github.com>	2024-10-27 19:03:47 +01:00
I-Al-Istannen	c54c3bcfa1	Fix crawling of favorites	2024-10-27 10:50:59 +01:00
I-Al-Istannen	d7f2229978	Bump version to 3.6.0	2024-10-23 20:17:47 +02:00
I-Al-Istannen	52fdeae752	Crawl custom item groups as folders	2024-10-21 23:43:48 +02:00
I-Al-Istannen	f9bb2e41cf	Sanitize slashes in exercise container names	2024-10-21 22:30:16 +02:00
Mr. Pine	4f9e2ab48d	Support named capture groups in regex transformers (#94 )	2024-10-21 15:21:33 +02:00
I-Al-Istannen	19beb8f07b	Document course overview downloading in config.md	2024-07-31 22:02:43 +02:00
I-Al-Istannen	c897d9e2f5	Support finding entries for course overview page Related to issue #93	2024-06-26 16:54:07 +02:00
Joscha	21a266e302	Update upload-artifact action to v4 https://github.com/actions/upload-artifact/blob/main/docs/MIGRATION.md#multiple-uploads-to-the-same-named-artifact	2024-05-11 16:33:14 +02:00
Pavel Zwerschke	b29b6f93f8	run ci twice Co-authored-by: Garmelon <joscha@plugh.de>	2024-05-11 16:09:46 +02:00
Pavel Zwerschke	318226d7cb	fix bump-version script	2024-05-11 10:27:54 +02:00
Pavel Zwerschke	422cf05f15	Move all configuration into pyproject.toml, add x86 mac to CI	2024-05-11 10:26:19 +02:00
Joscha	819c6673c7	Update changelog	2024-05-10 14:40:25 +02:00
Joscha	89b44c69a7	Update docs All config file options must be documented in CONFIG.md. The README.md is just a starting point. To avoid duplicated info, I've moved most of the docs to CONFIG.md.	2024-05-10 14:36:01 +02:00
Joscha	4b4f72b2ca	Fix command name	2024-05-10 14:34:20 +02:00
Joscha	778517d8c6	Fix KIT crawler requiring base_url and client_id options	2024-05-10 14:12:45 +02:00
Joscha	428b0179fc	Remove IliasConfig Also uses urljoin() in a few places that previously used string concatenation or fstrings. At this point, there isn't yet a need for IliasConfig, so I'd rather keep the code base simpler and more consistent. Should we need a structure like IliasConfig in the future (maybe because we have a few more ilias parsers), it's easy to add back.	2024-05-10 14:09:14 +02:00
Philipp Fruck	ade6309dd9	Update copyright information	2024-05-05 02:34:26 +02:00
Philipp Fruck	fd6cb7b966	docs: Remove some filler words	2024-05-05 02:34:00 +02:00
Philipp Fruck	5c87517ceb	docs: Explain usage with generic ilias	2024-05-04 17:52:12 +02:00
Philipp Fruck	b01f093474	fix: Element detection for other universities Other universities might use other URL schemes for different element types	2024-05-04 17:52:06 +02:00
Philipp Fruck	3a05b90525	fix circular import for _io_repeat	2024-05-04 17:51:59 +02:00
Philipp Fruck	7a00f73e0e	feat: Add authentication to generic ilias dl	2024-05-04 17:51:38 +02:00
Philipp Fruck	5d0621420e	feat: Generic ilias_web command	2024-05-04 17:44:37 +02:00
Philipp Fruck	df98153169	refactor: Extract generic settings from ilias command Preparation for generic ilias_web command	2024-05-04 17:44:30 +02:00
Philipp Fruck	fc1f68ccd9	refactor: Separate generic and KIT ilias functions	2024-05-04 17:44:18 +02:00
I-Al-Istannen	3e831c7e23	Fix normalization of meeting names in cards	2024-04-24 22:32:26 +02:00
Florian Raith	bbcfe9c8dd	Fix typo in CONFIG.md (#89 )	2024-04-19 16:52:18 +02:00
Joscha	eb01aa86cb	Bump version to 3.5.2	2024-04-14 12:10:17 +02:00
I-Al-Istannen	3db186a978	Fix personal desktop crawling HTML warnings	2024-04-10 11:15:25 +02:00
I-Al-Istannen	4a5959fd58	Fix personal desktop crawling without favorites	2024-04-10 11:15:25 +02:00
I-Al-Istannen	1cbc2b717a	Fix personal desktop crawling with ILIAS 8	2024-04-10 01:20:37 +02:00
Joscha	da627ff929	Bump version to 3.5.1	2024-04-09 14:28:56 +02:00
I-Al-Istannen	c1b592ac29	Fix ILIAS 8 file downloads truncating to zero bytes	2024-04-08 17:59:41 +02:00
I-Al-Istannen	eb0c956d32	Add compatibility with ILIAS 8	2024-04-05 19:08:05 +02:00
TornaxO7	ab0cb2d956	nix: bump nixpgs dependency	2024-02-27 23:39:53 +01:00
I-Al-Istannen	a117126389	Fix video name deduplication	2023-12-09 23:08:42 +01:00
I-Al-Istannen	e9f8901520	Fix typos in ilias crawler and use set literals	2023-11-30 20:57:57 +01:00
I-Al-Istannen	266812f90e	Move is_logged_in helper to kit_ilias_html	2023-11-16 11:19:20 +01:00
I-Al-Istannen	533bc27439	Bump version to 3.5.0	2023-09-13 23:13:30 +02:00
I-Al-Istannen	0113a0ca10	Update flake.lock	2023-09-13 22:23:36 +02:00
I-Al-Istannen	40f8a05ad6	Add .idea to gitignore	2023-09-13 22:23:36 +02:00
I-Al-Istannen	50b50513c6	Ignore SCORM learning modules	2023-08-29 13:51:19 +02:00
I-Al-Istannen	df3514cd03	Crawl paginated past meetings	2023-08-29 12:41:21 +02:00
I-Al-Istannen	ad53185247	Sanitize ascii control characters on windows	2023-08-29 12:41:15 +02:00
I-Al-Istannen	87b67e9271	Crawl files in the info tab	2023-08-29 12:41:15 +02:00
I-Al-Istannen	b54b3b979c	Remove size suffix for content pages	2023-08-27 11:43:05 +02:00
I-Al-Istannen	2184ac8040	Add support for ILIAS mediacast listings	2023-08-27 11:43:05 +02:00
I-Al-Istannen	b3d412360b	Add Nix flake	2023-08-26 23:54:19 +02:00
Mr. Pine	dbc2553b11	Add default `show-not-deleted` option If set to `no`, PFERD won't print status or report messages for not deleted files	2023-08-26 18:43:01 +02:00
I-Al-Istannen	68c398f1fe	Add support for ILIAS learning modules	2023-08-02 13:34:54 +02:00
I-Al-Istannen	123a57beec	Fix mypy unreachable error in file_templates	2023-07-29 18:36:33 +02:00
I-Al-Istannen	d204dac8ce	Detect unexpected root page redirects and abort operation	2023-07-29 18:36:33 +02:00
Mr. Pine	443f7fe839	Add `no-delete-prompt-overwrite` crawler conflict resolution option (#75 )	2023-07-29 18:36:33 +02:00
I-Al-Istannen	0294ceb7d5	Update github action versions	2023-03-22 00:10:54 +01:00
I-Al-Istannen	6f30c6583d	Fix crawling of cards without descriptions	2023-03-21 23:52:33 +01:00
I-Al-Istannen	467fc526e8	Fix crawling of file/video cards	2023-03-21 23:52:24 +01:00
I-Al-Istannen	722d2eb393	Fix crawling of courses with preselected timeline tab	2023-03-21 23:36:47 +01:00
Joscha	6d44aac278	Bump version to 3.4.3	2022-11-29 18:22:19 +01:00
c0derMo	55a2de6b88	Fix crawling English opencast	2022-11-29 18:13:56 +01:00
Joscha	c0d6d8b229	Use url after redirect for relative links	2022-11-21 18:10:45 +01:00
Joscha	635caa765d	Fix typo Thanks, burg113	2022-11-15 17:17:57 +01:00
Pavel Zwerschke	e69b55b349	Add more unofficial package managers (#66 )	2022-11-04 12:18:26 +01:00
Joscha	07200bbde5	Document ilias web crawler's forums option	2022-10-31 14:12:27 +01:00
I-Al-Istannen	c020cccc64	Include found paths in "second path found" warning	2022-10-29 14:08:29 +02:00
Joscha	259cfc20cc	Bump version to 3.4.2	2022-10-26 18:26:17 +02:00
Joscha	37b51a66d8	Update changelog	2022-10-26 18:22:37 +02:00
I-Al-Istannen	f47d2f11d8	Append trailing slash to kit-ipd links to ensure urljoin works as expected	2022-10-25 20:28:22 +02:00
I-Al-Istannen	1b6be6bd79	Handle content pages in cards	2022-10-24 18:37:26 +02:00
I-Al-Istannen	e1430e6298	Handle (and ignore) surveys	2022-10-24 18:37:26 +02:00
I-Al-Istannen	5fdd40204b	Unwrap future meetings when ILIAS hides them behind a pagination	2022-10-24 14:33:58 +02:00
I-Al-Istannen	fb4631ba18	Fix ilias background login	2022-10-24 13:13:36 +02:00
I-Al-Istannen	d72fc2760b	Handle empty forums	2022-10-24 13:12:17 +02:00
I-Al-Istannen	4a51aaa4f5	Fix forum crawling crashing for empty threads	2022-10-19 22:59:33 +02:00
Joscha	66a5b1ba02	Bump version to 3.4.1	2022-08-17 13:24:01 +02:00
I-Al-Istannen	aa5a3a10bc	Adjust changelog	2022-08-14 21:48:59 +02:00
I-Al-Istannen	d9b111cec2	Correctly nest description entries	2022-08-14 21:45:33 +02:00
I-Al-Istannen	345f52a1f6	Detect new login button	2022-08-14 21:41:29 +02:00
Joscha	ed24366aba	Add pass authenticator	2022-06-05 10:04:42 +02:00
I-Al-Istannen	46fb782798	Add forum crawling This downloads all forum posts when needed and saves each thread in its own html file, named after the thread title.	2022-05-24 23:43:53 +02:00
I-Al-Istannen	846c29aee1	Download page descriptions	2022-05-11 21:16:56 +02:00
I-Al-Istannen	a5015fe9b1	Correctly parse day-only meeting dates I failed to recognize the correct format in the previous adjustment, so this (hopefully) fixes it for good. Meetings apparently don't always have a time portion.	2022-05-08 23:22:26 +02:00
Joscha	616b0480f7	Simplify IPD crawler link regex	2022-05-08 18:18:05 +02:00
I-Al-Istannen	2f0e04ce13	Adjust changelog	2022-05-05 22:57:55 +02:00
I-Al-Istannen	bcc537468c	Fix crawling of expanded meetings The last meeting on every page is expanded by default. Its content is then shown inline and in the meeting page itself. We should skip the inline content.	2022-05-05 22:53:37 +02:00
I-Al-Istannen	694ffb4d77	Fix meeting date parsing Apparently the new pattern "<relative time qualifier>: <date>," was added. This patch adds support for it.	2022-05-05 22:28:30 +02:00
Joscha	af2cc1169a	Mention href for users of link_regex option	2022-05-05 14:36:03 +02:00
Joscha	bc3fa36637	Fix IPD crawler crashing on weird HTML comments	2022-05-05 14:35:42 +02:00
Joscha	afbd03f777	Fix docs	2022-05-05 14:35:42 +02:00
I-Al-Istannen	b8fe25c580	Add `.cpp` to ipd link regex	2022-05-04 14:19:26 +02:00
Joscha	a241672726	Bump version to 3.4.0	2022-05-01 22:29:06 +02:00
Joscha	a8f76e9be7	Use utf-8 for credential file	2022-04-29 23:15:12 +02:00
Joscha	b56475450d	Use utf-8 for cookies	2022-04-29 23:12:41 +02:00
Joscha	aa74604d29	Use utf-8 for report	2022-04-29 23:11:27 +02:00
Joscha	d2e6d91880	Make PFERD executable via python -m	2022-04-27 22:52:50 +02:00
Joscha	602044ff1b	Fix mypy errors and add missing await	2022-04-27 22:52:50 +02:00
Joscha	31631fb409	Increase minimum python version to 3.9	2022-04-27 22:52:50 +02:00
I-Al-Istannen	00db348218	Update changelog	2022-04-27 22:03:52 +02:00
I-Al-Istannen	a709280cbf	Try to detect unsupported config file encoding The encoding detection is quite rudimentary, but should detect the default windows encoding in many cases.	2022-04-27 22:03:47 +02:00
I-Al-Istannen	a99ddaa0cc	Read and write config in UTF-8	2022-04-27 21:47:51 +02:00
Joscha	ba3d299c05	Fix changelog	2022-04-27 21:26:24 +02:00
Joscha	07a21f80a6	Link to unofficial packages	2022-04-27 21:15:33 +02:00
I-Al-Istannen	f17b9b68f4	Add shibboleth authentication fix to changelog	2022-04-27 14:01:40 +02:00
I-Al-Istannen	a2831fbea2	Fix shib authentication Authentication failed previously if the shib session was still valid. If Shibboleth gets a request and the session is still valid, it directly responds without a second redirect.	2022-04-27 13:55:24 +02:00
I-Al-Istannen	da72863b47	Placate newer mypy	2022-04-03 13:19:08 +02:00
I-Al-Istannen	86e2e226dc	Notify user when shibboleth presents new entitlements	2022-04-03 11:37:08 +02:00
I-Al-Istannen	7872fe5221	Fix tables with more columns than expected	2022-01-18 22:38:48 +01:00
Joscha	86947e4874	Bump version to 3.3.1	2022-01-15 15:11:22 +01:00
Joscha	4f022e2d19	Reword changelog	2022-01-15 15:06:02 +01:00
I-Al-Istannen	f47e7374d2	Use fixed windows path for video cache	2022-01-15 12:00:30 +01:00
I-Al-Istannen	57ec51e95a	Fix login after shib url parser change	2022-01-14 20:17:27 +01:00
Joscha	0045124a4e	Bump version to 3.3.0	2022-01-09 21:09:09 +01:00
I-Al-Istannen	9618aae83b	Add content pages to changelog	2022-01-09 18:32:58 +01:00
I-Al-Istannen	33453ede2d	Update dependency versions in setup.py	2022-01-09 18:31:42 +01:00
I-Al-Istannen	e467b38d73	Only reject 1970 timestamps on windows	2022-01-09 18:23:00 +01:00
I-Al-Istannen	e9d2d05030	Update changelog	2022-01-09 11:48:26 +01:00
I-Al-Istannen	4bf0c972e6	Update types for rich 11	2022-01-09 11:48:26 +01:00
I-Al-Istannen	4ee919625d	Add rudimentary support for content pages	2022-01-08 20:47:35 +01:00
I-Al-Istannen	d30f25ee97	Detect shib login page as login page And do not assume we are logged in...	2022-01-08 20:28:45 +01:00
I-Al-Istannen	10d9d74528	Bail out when crawling recursive courses	2022-01-08 20:28:30 +01:00
I-Al-Istannen	43c5453e10	Correctly crawl files on desktop The files on the desktop do not include a download link, so we need to rewrite it.	2022-01-08 20:00:53 +01:00
I-Al-Istannen	eb4de8ae0c	Ignore 1970 dates as windows crashes when calling .timestamp()	2022-01-08 18:14:43 +01:00
I-Al-Istannen	e32c1f000f	Fix mtime for single streams	2022-01-08 18:05:48 +01:00
I-Al-Istannen	5f527bc697	Remove Python 3.9 Pattern typehints	2022-01-08 17:14:40 +01:00
I-Al-Istannen	ced8b9a2d0	Fix some accordions	2022-01-08 16:58:30 +01:00
I-Al-Istannen	6f3cfd4396	Fix personal desktop crawling	2022-01-08 16:58:15 +01:00
I-Al-Istannen	462d993fbc	Fix local video path cache (hopefully)	2022-01-08 00:27:48 +01:00
I-Al-Istannen	a99356f2a2	Fix video stream extraction	2022-01-08 00:27:34 +01:00
I-Al-Istannen	eac2e34161	Fix is_logged_in for ILIAS 7	2022-01-07 23:32:31 +01:00
I-Al-Istannen	a82a0b19c2	Collect crawler warnings/errors and include them in the report	2021-11-07 21:48:55 +01:00
I-Al-Istannen	90cb6e989b	Do not download single videos if cache does not exist	2021-11-06 23:21:15 +01:00
I-Al-Istannen	6289938d7c	Do not stop crawling files when encountering a CrawlWarning	2021-11-06 12:09:51 +01:00
I-Al-Istannen	13b8c3d9c6	Add regex option to config and CLI parser	2021-11-02 09:30:46 +01:00
I-Al-Istannen	88afe64a92	Refactor IPD crawler a bit	2021-11-02 01:25:01 +00:00
Julius Rüberg	6b2a657573	Fix IPD crawler for different subpages (#42 ) This patch reworks the IPD crawler to support subpages which do not use "/intern" for links and fetches the folder names from table headings.	2021-11-02 01:25:01 +00:00
Toorero	d6f38a61e1	Fixed minor spelling mistakes	2021-11-02 01:54:00 +01:00
I-Al-Istannen	ad3f4955f7	Update changelog	2021-10-30 18:14:39 +02:00
I-Al-Istannen	e42ab83d32	Add support for ILIAS cards	2021-10-30 18:13:44 +02:00
I-Al-Istannen	f9a3f9b9f2	Handle multi-stream videos	2021-10-30 18:12:29 +02:00
I-Al-Istannen	ef7d5ea2d3	Allow storing crawler-specific data in reports	2021-10-30 18:09:05 +02:00
lukasprobst	55ea304ff3	Disable interpolation of ConfigParser	2021-10-25 23:37:42 +02:00
Joscha	fee12b3d9e	Fix changelog	2021-10-25 17:44:12 +00:00
I-Al-Istannen	6673077397	Add kit-ipd crawler	2021-10-21 13:20:21 +02:00
Joscha	742632ed8d	Bump version to 3.2.0	2021-08-04 18:27:26 +00:00
Joscha	544d45cbc5	Catch non-critical exceptions at crawler top level	2021-07-13 15:42:11 +02:00
Joscha	86f79ff1f1	Update changelog	2021-07-07 15:23:58 +02:00
I-Al-Istannen	ee67f9f472	Sort elements by ILIAS id to ensure deterministic ordering	2021-07-06 17:45:48 +02:00
I-Al-Istannen	8ec3f41251	Crawl ilias booking objects as links	2021-07-06 16:15:25 +02:00
I-Al-Istannen	89be07d4d3	Use final crawl path in HTML parsing message	2021-07-03 17:05:48 +02:00
I-Al-Istannen	91200f3684	Fix nondeterministic name deduplication	2021-07-03 12:09:55 +02:00
Joscha	9ffd603357	Error when using multiple segments with -name-> Previously, PFERD just silently never matched the -name-> arrow. Now, it errors when loading the config file.	2021-07-01 11:14:50 +02:00
Joscha	80eeb8fe97	Add --skip option	2021-07-01 11:02:21 +02:00