pferd/CHANGELOG.md

# Changelog

All notable changes to this project will be documented in this file. The format
is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

This project has its own custom versioning scheme. Version numbers consist of
three parts (e. g. `3.1.5`).
- The first number is increased on major rewrites or changes. What classifies as
  a major change is up to the maintainers. This is pretty rare and a PFERD
  version 4 should hopefully not be necessary.
- The second number is increased on backwards-incompatible changes in behaviour.
  This refers to any change that would make an existing setup behave differently
  (e. g. renaming options or changing crawler behaviour). If this number is
  increased, it may be necessary for you to adapt your own setup.
- The third number is increased on backwards-compatible changes (e. g. adding
  new options or commands, changing documentation, fixing bugs). Updates that
  only increase this number should be safe and not require manual intervention.

We will try to correctly classify changes as backwards-compatible or
backwards-incompatible, but may occasionally make mistakes or stumble across
ambiguous situations.

## Unreleased

### Fixed
- Shibboleth login. It was broken due to URL parser changes and really
  *unfortunate* behaviour by aiohttp.
- local video cache on windows if the path was changed to accomodate windows
  file system limitations (e.g. replace `:`)

## 3.3.0 - 2022-01-09

### Added
- A KIT IPD crawler
- Support for ILIAS cards
- (Rudimentary) support for content pages
- Support for multi-stream videos
- Support for ILIAS 7

### Removed
- [Interpolation](https://docs.python.org/3/library/configparser.html#interpolation-of-values) in config file

### Fixed
- Crawling of recursive courses
- Crawling files directly placed on the personal desktop
- Ignore timestamps at the unix epoch as they crash on windows

## 3.2.0 - 2021-08-04

### Added
- `--skip` command line option
- Support for ILIAS booking objects

### Changed
- Using multiple path segments on left side of `-name->` now results in an
  error. This was already forbidden by the documentation but silently accepted
  by PFERD.
- More consistent path printing in some `--explain` messages

### Fixed
- Nondeterministic name deduplication due to ILIAS reordering elements
- More exceptions are handled properly

## 3.1.0 - 2021-06-13

If your config file doesn't do weird things with transforms, it should continue
to work. If your `-re->` arrows behave weirdly, try replacing them with
`-exact-re->` arrows. If you're on Windows, you might need to switch from `\`
path separators to `/` in your regex rules.

### Added
- `skip` option for crawlers
- Rules with `>>` instead of `>` as arrow head
- `-exact-re->` arrow (behaves like `-re->` did previously)

### Changed
- The `-re->` arrow can now rename directories (like `-->`)
- Use `/` instead of `\` as path separator for (regex) rules on Windows
- Use the label to the left for exercises instead of the button name to
  determine the folder name

### Fixed
- Video pagination handling in ILIAS crawler

## 3.0.1 - 2021-06-01

### Added
- `credential-file` authenticator
- `--credential-file` option for `kit-ilias-web` command
- Warning if using concurrent tasks with `kit-ilias-web`

### Changed
- Cookies are now stored in a text-based format

### Fixed
- Date parsing now also works correctly in non-group exercises

## 3.0.0 - 2021-05-31

### Added
- Proper config files
- Concurrent crawling
- Crawl external ILIAS links
- Crawl uploaded exercise solutions
- Explain what PFERD is doing and why (`--explain`)
- More control over output (`--status`, `--report`)
- Debug transform rules with `--debug-transforms`
- Print report after exiting via Ctrl+C
- Store crawler reports in `.report` JSON file
- Extensive config file documentation (`CONFIG.md`)
- Documentation for developers (`DEV.md`)
- This changelog

### Changed
- Rewrote almost everything
- Better error messages
- Redesigned CLI
- Redesigned transform rules
- ILIAS crawling logic (paths may be different)
- Better support for weird paths on Windows
- Set user agent (`PFERD/<version>`)

### Removed
- Backwards compatibility with 2.x
- Python files as config files
- Some types of crawlers
Add changelog 2021-04-29 16:14:50 +02:00			`# Changelog`

Document versioning scheme 2021-05-29 21:38:36 +02:00			`All notable changes to this project will be documented in this file. The format`
			`is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).`
Add changelog 2021-04-29 16:14:50 +02:00
Document versioning scheme 2021-05-29 21:38:36 +02:00			`This project has its own custom versioning scheme. Version numbers consist of`
			three parts (e. g. `3.1.5`).
			`- The first number is increased on major rewrites or changes. What classifies as`
			`a major change is up to the maintainers. This is pretty rare and a PFERD`
			`version 4 should hopefully not be necessary.`
			`- The second number is increased on backwards-incompatible changes in behaviour.`
			`This refers to any change that would make an existing setup behave differently`
			`(e. g. renaming options or changing crawler behaviour). If this number is`
			`increased, it may be necessary for you to adapt your own setup.`
			`- The third number is increased on backwards-compatible changes (e. g. adding`
			`new options or commands, changing documentation, fixing bugs). Updates that`
			`only increase this number should be safe and not require manual intervention.`

			`We will try to correctly classify changes as backwards-compatible or`
			`backwards-incompatible, but may occasionally make mistakes or stumble across`
			`ambiguous situations.`
Add changelog 2021-04-29 16:14:50 +02:00
Add script for releasing new versions 2021-05-24 20:08:49 +02:00			`## Unreleased`
Add changelog 2021-04-29 16:14:50 +02:00
Fix login after shib url parser change 2022-01-14 20:15:19 +01:00			`### Fixed`
Use fixed windows path for video cache 2022-01-14 22:01:45 +01:00			`- Shibboleth login. It was broken due to URL parser changes and really`
Fix login after shib url parser change 2022-01-14 20:15:19 +01:00			`unfortunate behaviour by aiohttp.`
Use fixed windows path for video cache 2022-01-14 22:01:45 +01:00			`- local video cache on windows if the path was changed to accomodate windows`
			file system limitations (e.g. replace `:`)
Fix login after shib url parser change 2022-01-14 20:15:19 +01:00
Bump version to 3.3.0 2022-01-09 21:09:09 +01:00			`## 3.3.0 - 2022-01-09`

Fix changelog 2021-10-25 19:44:12 +02:00			`### Added`
			`- A KIT IPD crawler`
Update changelog 2021-10-30 18:14:39 +02:00			`- Support for ILIAS cards`
Add content pages to changelog 2022-01-09 18:32:58 +01:00			`- (Rudimentary) support for content pages`
Update changelog 2021-10-30 18:14:39 +02:00			`- Support for multi-stream videos`
Update changelog 2022-01-09 11:39:42 +01:00			`- Support for ILIAS 7`
Fix changelog 2021-10-25 19:44:12 +02:00
Disable interpolation of ConfigParser 2021-10-25 22:32:54 +02:00			`### Removed`
			`- [Interpolation](https://docs.python.org/3/library/configparser.html#interpolation-of-values) in config file`

Update changelog 2022-01-09 11:39:42 +01:00			`### Fixed`
			`- Crawling of recursive courses`
			`- Crawling files directly placed on the personal desktop`
			`- Ignore timestamps at the unix epoch as they crash on windows`

Bump version to 3.2.0 2021-08-04 20:27:26 +02:00			`## 3.2.0 - 2021-08-04`

Update changelog 2021-07-07 14:26:20 +02:00			`### Added`
			- `--skip` command line option
			`- Support for ILIAS booking objects`

			`### Changed`
			- Using multiple path segments on left side of `-name->` now results in an
			`error. This was already forbidden by the documentation but silently accepted`
			`by PFERD.`
			- More consistent path printing in some `--explain` messages

			`### Fixed`
			`- Nondeterministic name deduplication due to ILIAS reordering elements`
Catch non-critical exceptions at crawler top level 2021-07-13 15:42:11 +02:00			`- More exceptions are handled properly`
Update changelog 2021-07-07 14:26:20 +02:00
Bump version to 3.1.0 2021-06-13 17:23:18 +02:00			`## 3.1.0 - 2021-06-13`

Add migration notes to changelog Also clean up some other formatting for consistency 2021-06-13 15:06:50 +02:00			`If your config file doesn't do weird things with transforms, it should continue`
			to work. If your `-re->` arrows behave weirdly, try replacing them with
			`-exact-re->` arrows. If you're on Windows, you might need to switch from `\`
			path separators to `/` in your regex rules.

Add 'skip' option to crawlers 2021-06-04 18:33:02 +02:00			`### Added`
			- `skip` option for crawlers
Overhaul transform logic -re-> arrows now rename their parent directories (like -->) and don't require a full match (like -exact->). Their old behaviour is available as -exact-re->. Also, this change adds the ">>" arrow head, which modifies the current path and continues to the next rule when it matches. 2021-06-09 17:42:38 +02:00			- Rules with `>>` instead of `>` as arrow head
			- `-exact-re->` arrow (behaves like `-re->` did previously)
Add 'skip' option to crawlers 2021-06-04 18:33:02 +02:00
Always use '/' as path separator for regex rules Previously, regex-matching paths on windows would, in some cases, require four backslashes ('\\\\') to escape a single path separator. That's just too much. With this commit, regex transforms now use '/' instead of '\' as path separator, meaning rules can more easily be shared between platforms (although they are not guaranteed to be 100% compatible since on Windows, '\' is still recognized as a path separator). To make rules more intuitive to write, local relative paths are now also printed with '/' as path separator on Windows. Since Windows also accepts '/' as path separator, this change doesn't really affect other rules that parse their sides as paths. 2021-06-04 18:02:45 +02:00			`### Changed`
Overhaul transform logic -re-> arrows now rename their parent directories (like -->) and don't require a full match (like -exact->). Their old behaviour is available as -exact-re->. Also, this change adds the ">>" arrow head, which modifies the current path and continues to the next rule when it matches. 2021-06-09 17:42:38 +02:00			- The `-re->` arrow can now rename directories (like `-->`)
Always use '/' as path separator for regex rules Previously, regex-matching paths on windows would, in some cases, require four backslashes ('\\\\') to escape a single path separator. That's just too much. With this commit, regex transforms now use '/' instead of '\' as path separator, meaning rules can more easily be shared between platforms (although they are not guaranteed to be 100% compatible since on Windows, '\' is still recognized as a path separator). To make rules more intuitive to write, local relative paths are now also printed with '/' as path separator on Windows. Since Windows also accepts '/' as path separator, this change doesn't really affect other rules that parse their sides as paths. 2021-06-04 18:02:45 +02:00			- Use `/` instead of `\` as path separator for (regex) rules on Windows
Use the exercise label instead of the button name as path 2021-06-04 19:23:33 +02:00			`- Use the label to the left for exercises instead of the button name to`
			`determine the folder name`
Always use '/' as path separator for regex rules Previously, regex-matching paths on windows would, in some cases, require four backslashes ('\\\\') to escape a single path separator. That's just too much. With this commit, regex transforms now use '/' instead of '\' as path separator, meaning rules can more easily be shared between platforms (although they are not guaranteed to be 100% compatible since on Windows, '\' is still recognized as a path separator). To make rules more intuitive to write, local relative paths are now also printed with '/' as path separator on Windows. Since Windows also accepts '/' as path separator, this change doesn't really affect other rules that parse their sides as paths. 2021-06-04 18:02:45 +02:00
Fix wrong base URL for multi-stage pages 2021-06-13 15:39:22 +02:00			`### Fixed`
			`- Video pagination handling in ILIAS crawler`

Bump version to 3.0.1 2021-06-01 11:49:30 +02:00			`## 3.0.1 - 2021-06-01`

Add credential file authenticator 2021-05-31 17:55:56 +02:00			`### Added`
			- `credential-file` authenticator
Add CLI option for credential file auth to kit-ilias-web 2021-05-31 18:19:05 +02:00			- `--credential-file` option for `kit-ilias-web` command
Warn if using concurrent tasks with kit-ilias-web 2021-05-31 21:07:13 +02:00			- Warning if using concurrent tasks with `kit-ilias-web`
Add credential file authenticator 2021-05-31 17:55:56 +02:00
Store cookies in text-based format Using the stdlib's http.cookie module, cookies are now stored as one "Set-Cookie" header per line. Previously, the aiohttp.CookieJar's save() and load() methods were used (which use pickling). 2021-05-31 22:04:56 +02:00			`### Changed`
			`- Cookies are now stored in a text-based format`

Fix exercise date parsing for non-group submissions ILIAS apparently changes the order of the fields as it sees fit, so we now try to parse every column, starting at from the right, as a date. The first column that parses successfully is then used. 2021-05-31 18:00:42 +02:00			`### Fixed`
			`- Date parsing now also works correctly in non-group exercises`

Bump version to 3.0.0 2021-05-31 12:49:04 +02:00			`## 3.0.0 - 2021-05-31`

Add changelog 2021-04-29 16:14:50 +02:00			`### Added`
Update README, CHANGELOG 2021-05-25 17:16:57 +02:00			`- Proper config files`
			`- Concurrent crawling`
			`- Crawl external ILIAS links`
			`- Crawl uploaded exercise solutions`
			- Explain what PFERD is doing and why (`--explain`)
			- More control over output (`--status`, `--report`)
Document --debug-transforms 2021-05-26 11:47:51 +02:00			- Debug transform rules with `--debug-transforms`
Update README, CHANGELOG 2021-05-25 17:16:57 +02:00			`- Print report after exiting via Ctrl+C`
			- Store crawler reports in `.report` JSON file
			- Extensive config file documentation (`CONFIG.md`)
			- Documentation for developers (`DEV.md`)
Add changelog 2021-04-29 16:14:50 +02:00			`- This changelog`

			`### Changed`
			`- Rewrote almost everything`
Update README, CHANGELOG 2021-05-25 17:16:57 +02:00			`- Better error messages`
Add changelog 2021-04-29 16:14:50 +02:00			`- Redesigned CLI`
Update README, CHANGELOG 2021-05-25 17:16:57 +02:00			`- Redesigned transform rules`
			`- ILIAS crawling logic (paths may be different)`
			`- Better support for weird paths on Windows`
			- Set user agent (`PFERD/<version>`)
Add changelog 2021-04-29 16:14:50 +02:00
			`### Removed`
			`- Backwards compatibility with 2.x`
Update README, CHANGELOG 2021-05-25 17:16:57 +02:00			`- Python files as config files`
			`- Some types of crawlers`