mirrors/pferd
1
0
Fork 0
mirror of https://github.com/Garmelon/PFERD.git synced 2025-09-02 03:22:27 +02:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • c454fabc9d Add support for exercises in ILIAS crawler I-Al-Istannen 2021-05-15 21:40:17 +02:00
  • 7d323ec62b Implement video downloads in ilias crawler I-Al-Istannen 2021-05-15 21:29:43 +02:00
  • c7494e32ce Start implementing crawling in ILIAS crawler I-Al-Istannen 2021-05-15 20:42:18 +02:00
  • 1123c8884d Implement an IliasPage I-Al-Istannen 2021-05-15 18:57:17 +02:00
  • e1104f888d Add tfa authenticator Joscha 2021-05-15 18:27:16 +02:00
  • 8c32da7f19 Let authenticators provide username and password separately Joscha 2021-05-15 18:24:03 +02:00
  • d63494908d Properly invalidate exceptions Joscha 2021-05-15 17:37:05 +02:00
  • b70b62cef5 Make crawler sections start with "crawl:" Joscha 2021-05-15 17:23:33 +02:00
  • 868f486922 Rename local crawler path to target Joscha 2021-05-15 17:12:25 +02:00
  • b2a2b5999b Implement ILIAS auth and crawl home page I-Al-Istannen 2021-05-15 15:18:51 +02:00
  • 595de88d96 Fix authenticator and crawler names Joscha 2021-05-15 15:18:16 +02:00
  • a6fdf05ee9 Allow variable whitespace in arrow rules Joscha 2021-05-15 15:13:34 +02:00
  • f897d7c2e1 Add name variants for all arrows Joscha 2021-05-15 15:06:45 +02:00
  • b0f731bf84 Make crawlers use transformers Joscha 2021-05-15 14:03:15 +02:00
  • 302b8c0c34 Fix errors loading local crawler config Joscha 2021-05-15 13:32:13 +02:00
  • acd674f0a0 Change limiter logic Joscha 2021-05-15 13:21:38 +02:00
  • b0f9e1e8b4 Add vscode directory to gitignore I-Al-Istannen 2021-05-15 11:20:20 +02:00
  • ed2e19a150 Add reasons for invalid values Joscha 2021-05-15 00:39:55 +02:00
  • 296a169dd3 Make limiter logic more complex Joscha 2021-05-15 00:38:46 +02:00
  • 1591cb9197 Add options to slow down local crawler Joscha 2021-05-14 21:41:24 +02:00
  • 0c9167512c Fix output dir Joscha 2021-05-14 21:28:38 +02:00
  • a673ab0fae Delete old files Joscha 2021-05-14 00:20:59 +02:00
  • 6e5fdf4e9e Set user agent to "pferd/<version>" Joscha 2021-05-14 00:09:58 +02:00
  • 93a5a94dab Single-source version number Joscha 2021-05-13 23:52:46 +02:00
  • d565df27b3 Add HttpCrawler Joscha 2021-05-13 22:28:14 +02:00
  • 961f40f9a1 Document simple authenticator Joscha 2021-05-13 19:55:04 +02:00
  • e3ee4e515d Disable highlighting of primitives Joscha 2021-05-13 19:47:44 +02:00
  • 94d6a01cca Use file mtime in local crawler Joscha 2021-05-13 19:42:40 +02:00
  • 38bb66a776 Update file metadata in more cases Joscha 2021-05-13 19:40:10 +02:00
  • 68781a88ab Fix asynchronous methods being not awaited Joscha 2021-05-13 19:39:49 +02:00
  • 910462bb72 Log stuff happening to files Joscha 2021-05-13 19:37:27 +02:00
  • 6bd6adb977 Fix tmp file names Joscha 2021-05-13 19:36:46 +02:00
  • 0acdee15a0 Let crawlers obtain authenticators Joscha 2021-05-13 18:57:20 +02:00
  • c3ce6bb31c Fix crawler cleanup not being awaited Joscha 2021-05-11 00:28:45 +02:00
  • 0459ed093e Add simple authenticator Joscha 2021-05-11 00:27:43 +02:00
  • d5f29f01c5 Use global conductor instance Joscha 2021-05-10 23:50:16 +02:00
  • 595ba8b7ab Remove dummy crawler Joscha 2021-05-10 23:47:46 +02:00
  • cec0a8e1fc Fix mymy errors Joscha 2021-05-09 01:45:01 +02:00
  • f9b2fd60e2 Document local crawler and auth Joscha 2021-05-09 01:33:47 +02:00
  • 60cd9873bc Add local file crawler Joscha 2021-05-06 01:02:40 +02:00
  • 273d56c39a Properly load crawler config Joscha 2021-05-05 23:45:10 +02:00
  • 5497dd2827 Add @noncritical and @repeat decorators Joscha 2021-05-05 23:36:54 +02:00
  • bbfdadc463 Implement output directory Joscha 2021-05-05 18:08:34 +02:00
  • fde811ae5a Document on_conflict option Joscha 2021-05-05 00:55:55 +02:00
  • 07e831218e Add sync report Joscha 2021-05-02 00:56:10 +02:00
  • 91c33596da Load crawlers from config file Joscha 2021-04-30 16:22:14 +02:00
  • a8dcf941b9 Document possible redownload settings Joscha 2021-04-30 15:32:56 +02:00
  • e7a51decb0 Elaborate on transforms and implement changes Joscha 2021-04-29 20:13:46 +02:00
  • 9ec19be113 Document config file format Joscha 2021-04-29 18:55:08 +02:00
  • f776186480 Use PurePath instead of Path Joscha 2021-04-29 16:52:00 +02:00
  • 0096d83387 Simplify Limiter implementation Joscha 2021-04-29 16:37:42 +02:00
  • 20a24dbcbf Add changelog Joscha 2021-04-29 16:14:50 +02:00
  • 502654d853 Fix mypy errors Joscha 2021-04-29 15:47:52 +02:00
  • d2103d7c44 Document crawler Joscha 2021-04-29 15:43:20 +02:00
  • d96a361325 Test and fix exclusive output Joscha 2021-04-29 15:26:10 +02:00
  • 2e85d26b6b Use conductor via context manager Joscha 2021-04-29 14:23:28 +02:00
  • 6431a3fb3d Fix some mypy errors Joscha 2021-04-29 14:23:09 +02:00
  • ac3bfd7388 Make progress bars easier to use Joscha 2021-04-29 13:53:16 +02:00
  • 3ea86d18a0 Jerry-rig DummyCrawler to run Joscha 2021-04-29 13:45:04 +02:00
  • bbc792f9fb Implement Crawler and DummyCrawler Joscha 2021-04-29 13:44:29 +02:00
  • 7e127cd5cc Clean up and fix conductor and limiter Joscha 2021-04-29 13:43:50 +02:00
  • c4fb92c658 Make type hints compatible with Python 3.8 Joscha 2021-04-29 13:11:58 +02:00
  • 8da1ac6cee Extend mypy config Joscha 2021-04-29 11:25:13 +02:00
  • a18db57e6f Implement terminal conductor Joscha 2021-04-29 11:25:00 +02:00
  • b915e393dd Implement limiter Joscha 2021-04-29 10:24:28 +02:00
  • 3a74c23d09 Implement transformer Joscha 2021-04-29 09:51:25 +02:00
  • fbebc46c58 Load and dump config Joscha 2021-04-27 12:41:49 +02:00
  • cccd68e04a Bump version to v2.6.2 v2.6.2 v2 I-Al-Istannen 2021-04-29 00:18:26 +02:00
  • 2bd40a5f30 Fix -p and -u flags I-Al-Istannen 2021-04-29 00:15:12 +02:00
  • 5595a908d8 Configure entry point Joscha 2021-04-27 00:29:42 +02:00
  • 27e4abcfa3 Do project setup from scratch Joscha 2021-04-26 23:46:44 +02:00
  • 2ca1101326 Fix typo in sync_url I-Al-Istannen 2021-04-19 14:53:16 +02:00
  • c1ab7485e2 Bump version to 2.6.1 v2.6.1 I-Al-Istannen 2021-04-19 11:21:56 +02:00
  • 29cd5d1a3c Reflect totality of sanitize_windows_path in return type I-Al-Istannen 2021-04-19 11:10:02 +02:00
  • 6d5d9333ad Force folder to be file-system path I-Al-Istannen 2021-04-19 11:07:25 +02:00
  • 7cc40595dc Allow synchronizing to directory "." I-Al-Istannen 2021-04-14 20:25:25 +02:00
  • 80ae5ddfaa Bump version to v2.6.0 v2.6.0 I-Al-Istannen 2021-04-14 19:47:41 +02:00
  • 4f480d117e Install keyring in CI I-Al-Istannen 2021-04-14 19:24:05 +02:00
  • 1f2af3a290 Retry on more I/O Errors I-Al-Istannen 2021-04-13 11:32:55 +02:00
  • 14cdfb6a69 Fix typo in date demangler doc I-Al-Istannen 2021-04-13 11:19:51 +02:00
  • e2bf84392b [sync_url] Properly declare "no-videos" as flag I-Al-Istannen 2021-04-08 18:12:27 +02:00
  • 946b7a7931 Also crawl .c/.java/.zip from IPD page I-Al-Istannen 2021-02-09 12:30:59 +01:00
  • 9a9018751e Bump version v2.5.4 I-Al-Istannen 2021-02-06 22:54:05 +01:00
  • 83b75e8254 syncurl: Sanitize element name on windows if it is used as folder name I-Al-Istannen 2021-02-06 22:51:08 +01:00
  • 35c3fa205d Fixed description of activating venv (#22) Toorero 2021-01-28 21:24:09 +01:00
  • 0b606f02fa Bump version v2.5.3 I-Al-Istannen 2021-01-17 10:33:10 +01:00
  • fb78a6e98e Retry ILIAS downloads a few times and only fail that file I-Al-Istannen 2021-01-06 12:29:24 +01:00
  • 5de68a0400 Bump version v2.5.2 I-Al-Istannen 2020-12-30 17:20:30 +01:00
  • f0562049b6 Remove Python 3.9 method in crawler I-Al-Istannen 2020-12-30 17:18:04 +01:00
  • 0e1077bb50 Bump version v2.5.1 I-Al-Istannen 2020-12-30 14:50:49 +01:00
  • c978e9edf4 Resolve a few pylint warnings I-Al-Istannen 2020-12-30 14:45:46 +01:00
  • 2714ac6be6 Send CSRF token to Shibboleth I-Al-Istannen 2020-12-30 14:34:11 +01:00
  • 9b048a9cfc Canonize meeting names to a properly formatted date I-Al-Istannen 2020-12-30 14:32:59 +01:00
  • 1c2b6bf994 Bump version v2.5.0 I-Al-Istannen 2020-12-13 19:57:29 +01:00
  • ee39aaf08b Fix merge marker in LICENSE I-Al-Istannen 2020-12-07 22:55:28 +01:00
  • 93e6329901 Use the least destructive conflict resolver if there are multiple I-Al-Istannen 2020-12-06 13:28:08 +01:00
  • f47b137b59 Fix ILIAS init.py and Pferd.py authenticators I-Al-Istannen 2020-12-05 23:35:20 +01:00
  • 83ea15ee83 Use system keyring service for password auth Scriptim 2020-11-04 00:18:27 +01:00
  • 75471c46d1 Use credential file I-Al-Istannen 2020-11-04 21:18:48 +01:00
  • 1e0343bba6 sync_url: Add username and password args Lucas 2020-11-03 13:38:33 +01:00