mirrors/pferd
1
0
Fork 0
mirror of https://github.com/Garmelon/PFERD.git synced 2023-12-21 10:23:01 +01:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • 086b15d10f Crawl a bit more iteratively I-Al-Istannen 2020-05-30 15:35:58 +02:00
  • 9d6ce331a5 Use IliasCrawlerEntry entries in the ilias scraper I-Al-Istannen 2020-05-30 15:04:54 +02:00
  • 821c7ade26 Move video url extraction logic to crawler I-Al-Istannen 2020-05-30 00:14:08 +02:00
  • b969a1854a Remove unneeded whitespace I-Al-Istannen 2020-05-30 00:22:17 +02:00
  • 62535b4452 Unpack videos in ILIAS downloader I-Al-Istannen 2020-05-21 21:57:31 +02:00
  • c0056e5669 Correctly crawl video pages with multiple pages I-Al-Istannen 2020-05-21 21:13:19 +02:00
  • cfe4a8fc0a Bump version to 2.0.0 v2.0.0 Joscha 2020-05-15 11:26:23 +00:00
  • 95b9248a25 Clean up Joscha 2020-05-15 11:26:09 +00:00
  • 1004fa40f8 Add personal desktop example config to README I-Al-Istannen 2020-05-15 11:02:55 +02:00
  • e8ddb0ca04 Fix example config link in README I-Al-Istannen 2020-05-15 11:02:45 +02:00
  • 36c8785f15 Add example config that synchronizes the personal desktop I-Al-Istannen 2020-05-15 11:02:13 +02:00
  • 03a801eecc Correctly type hint swallow_and_print_errors decorator I-Al-Istannen 2020-05-12 21:02:41 +02:00
  • 072c6630bf Avoid logging import in config Joscha 2020-05-12 18:19:23 +00:00
  • 4f56c8f192 Pass element type to ilias directory filter I-Al-Istannen 2020-05-12 14:38:25 +02:00
  • 4fdb67128d Fetch correct diva playlist id I-Al-Istannen 2020-05-11 00:25:34 +02:00
  • a0f9d31d94 Use PrettyLogger warning everywhere I-Al-Istannen 2020-05-10 21:53:24 +02:00
  • e7b08420ba Warn when a marked file is added again I-Al-Istannen 2020-05-10 21:37:48 +02:00
  • c1b21f7772 Only remove a progress task when we added it I-Al-Istannen 2020-05-10 12:28:30 +02:00
  • 9850ab1d73 Allow crawling the ILIAS Personal Desktop I-Al-Istannen 2020-05-10 12:16:42 +02:00
  • 9950144e97 Allow passing a playlist URL to diva instead of an id I-Al-Istannen 2020-05-10 11:11:28 +02:00
  • f6faacabb0 Move FatalException to errors.py I-Al-Istannen 2020-05-09 00:00:21 +02:00
  • 19c1e3ac6f Fail on invalid ILIAS course ids I-Al-Istannen 2020-05-08 23:47:05 +02:00
  • afa48c2d2d Swallow and print errors instead of crashing I-Al-Istannen 2020-05-08 23:35:25 +02:00
  • a4c518bf4c Update date find regex I-Al-Istannen 2020-05-08 22:17:58 +02:00
  • 057135022f Try to accept that life sometimes is in English I-Al-Istannen 2020-05-08 22:10:43 +02:00
  • 755e9aa0d3 Try to add support for Shibboleth TFA token I-Al-Istannen 2020-05-08 21:51:33 +02:00
  • c9deca19ca Remove walrus to lower needed python version I-Al-Istannen 2020-05-08 21:21:33 +02:00
  • bb048c3a6d Apparently we want Python 3.8 I-Al-Istannen 2020-05-08 21:03:49 +02:00
  • 33fcd307b2 Adjust install directions I-Al-Istannen 2020-05-08 20:53:41 +02:00
  • a0c5572b59 Fix progress bars swallowing a line when they shouldn't I-Al-Istannen 2020-05-08 19:55:53 +02:00
  • 2d20d2934c Color warning differently I-Al-Istannen 2020-05-08 19:48:21 +02:00
  • 2c48ab66d4 Use rich for log colorization I-Al-Istannen 2020-05-08 19:28:34 +02:00
  • 104b838aed Automatically discover packages in setup.py I-Al-Istannen 2020-05-08 18:55:30 +02:00
  • 7f10931be8 Add rich to setup.py I-Al-Istannen 2020-05-08 18:49:36 +02:00
  • 07c225bc20 Expand README. I did not proofread this thing :( I-Al-Istannen 2020-05-08 18:47:58 +02:00
  • 56f2394001 Add a download progress bar I-Al-Istannen 2020-05-08 00:26:33 +02:00
  • fdff8bc40e example_config: Change db transform I-Al-Istannen 2020-05-01 13:31:29 +02:00
  • bee3d70998 Added a diva playlist downloader I-Al-Istannen 2020-04-30 16:24:38 +02:00
  • 42345ecc61 Demangle "Morgen" too I-Al-Istannen 2020-04-30 12:00:21 +02:00
  • 920d521d68 Change PrettyLogger.warn to PrettyLogger.warning I-Al-Istannen 2020-04-25 20:11:51 +02:00
  • e0b46a306a Use warn method in IliasCrawler I-Al-Istannen 2020-04-25 20:05:19 +02:00
  • 8a42a2a396 Move logging into its own file I-Al-Istannen 2020-04-25 19:59:58 +02:00
  • 80247400a4 Debug log when starting an ilias download I-Al-Istannen 2020-04-25 13:02:07 +02:00
  • 13c5a29ff0 Fix and extend example config Joscha 2020-04-24 18:41:22 +00:00
  • 1aaa6e7ab5 Use PathLike everywhere Joscha 2020-04-24 18:39:30 +00:00
  • 7f53543324 Satisfy pylint and add todo Joscha 2020-04-24 18:26:28 +00:00
  • 292e516297 Change crawler and downloader output Joscha 2020-04-24 18:24:44 +00:00
  • 8258fa8919 Add test run option to PFERD Joscha 2020-04-24 18:00:21 +00:00
  • 5b929f09a2 Move download strategies to downloader Joscha 2020-04-24 14:26:20 +00:00
  • 4d32f863bc Clean up organizer after synchronizing Joscha 2020-04-24 14:17:23 +00:00
  • 4e7333b396 Allow specifying paths as strings in Pferd Joscha 2020-04-24 11:50:27 +00:00
  • 4c0e3b493a Use download_modified_or_new as default strategy I-Al-Istannen 2020-04-24 13:48:06 +02:00
  • 2de079a5d3 Add a few Transform combinators Joscha 2020-04-24 11:35:31 +00:00
  • 509e624d47 Satisfy pyling. Useful docstrings? Not quite sure. I-Al-Istannen 2020-04-23 20:31:32 +02:00
  • ca8fcf7a1d Somewhat elaborate example_config I-Al-Istannen 2020-04-23 20:22:41 +02:00
  • 980f69b5af Fix organizer marking itself causing an error I-Al-Istannen 2020-04-23 20:02:05 +02:00
  • 0b00a9c26b Log when starting to synchronize I-Al-Istannen 2020-04-23 19:55:37 +02:00
  • 1ef85c45e5 Switch Transform to PurePath Joscha 2020-04-23 17:38:41 +00:00
  • 5ef5a56e69 Extract Location into separate file Joscha 2020-04-23 17:38:28 +00:00
  • f3f4be2690 More free functions I-Al-Istannen 2020-04-23 19:18:21 +02:00
  • 076b8c5a1f Add download strategies to save bandwith I-Al-Istannen 2020-04-23 18:29:20 +02:00
  • 13bc78c889 Display reason for ignoring an element in ilias crawler I-Al-Istannen 2020-04-23 13:54:58 +02:00
  • dc964a9d98 Remove finished TODOs I-Al-Istannen 2020-04-23 13:30:34 +02:00
  • c2b14f3db9 ilias crawler: Use direct download link if possible I-Al-Istannen 2020-04-23 13:07:07 +02:00
  • 4b59a7c375 Move around TODOs Joscha 2020-04-23 10:49:01 +00:00
  • 3a57430893 Fix type errors in example_config I-Al-Istannen 2020-04-23 12:35:58 +02:00
  • bef210ae77 Rename and implement IliasDirectoryFilter I-Al-Istannen 2020-04-23 12:33:38 +02:00
  • ea005517cf Only remove folders if they exist in tmpdir I-Al-Istannen 2020-04-23 12:09:45 +02:00
  • 3841f27aab Add example config Joscha 2020-04-23 09:49:15 +00:00
  • df0eb84a44 Fix TmpDir and Location Joscha 2020-04-23 09:48:09 +00:00
  • 2de4255a78 Add Pferd class Joscha 2020-04-23 09:44:13 +00:00
  • 3c808879c9 Add Transforms and Transformables Joscha 2020-04-22 18:25:09 +00:00
  • a051e3bcca ilias crawler: Add some unhelpful documentation I-Al-Istannen 2020-04-22 17:58:19 +02:00
  • eb7df036df WIP: ilias crawler: Also crawl assignments I-Al-Istannen 2020-04-22 14:32:20 +02:00
  • 23db59e733 WIP: ilias-crawler: Demangle dates I-Al-Istannen 2020-04-22 12:44:41 +02:00
  • ac65b06a8e Satisfy pylint a bit I-Al-Istannen 2020-04-22 01:37:34 +02:00
  • 8891041069 WIP: crawler: Add opencast video crawler I-Al-Istannen 2020-04-21 23:01:19 +02:00
  • 70d63e3e90 WIP: Start small ILIAS crawler I-Al-Istannen 2020-04-21 13:32:03 +02:00
  • b2a7af2e3e Store modification_date in IliasDownloadInfo, remove parameters I-Al-Istannen 2020-04-21 13:31:50 +02:00
  • 23bed48c8c Satisfy autopep8 I-Al-Istannen 2020-04-21 13:30:42 +02:00
  • 0926d33798 Use downloader-specific data classes Joscha 2020-04-20 18:06:21 +00:00
  • 55ba2f4070 Fix pylint in downloaders I-Al-Istannen 2020-04-20 19:48:59 +02:00
  • d18b48aaf4 Stream in http downloader I-Al-Istannen 2020-04-20 19:45:25 +02:00
  • 4ef0ffe3bf Listen to pylint and mypy Joscha 2020-04-20 17:27:26 +00:00
  • ce77995c8f Rename http downloader module Joscha 2020-04-20 17:08:51 +00:00
  • ed9245c14d Remove old organizer I-Al-Istannen 2020-04-20 18:50:23 +02:00
  • 01e6972c96 Add ilias downloader I-Al-Istannen 2020-04-20 18:36:40 +02:00
  • 8181ae5b17 Guard http response in context manager I-Al-Istannen 2020-04-20 18:04:56 +02:00
  • 6407190ae0 Soupify requests responses properly Joscha 2020-04-20 16:38:18 +00:00
  • 87395faac2 Add base for simple HTTP downloader I-Al-Istannen 2020-04-20 17:43:41 +02:00
  • a9e6e7883d Create temp dir folder in constructor I-Al-Istannen 2020-04-20 17:43:28 +02:00
  • 154d6b29dd Listen to pylint Joscha 2020-04-20 15:15:47 +00:00
  • 62ac569ec4 Revert "Add proposed crawler entry type" I-Al-Istannen 2020-04-20 16:59:08 +02:00
  • 9f1a0a58ab Add proposed crawler entry type I-Al-Istannen 2020-04-20 16:54:32 +02:00
  • 879a2c7c80 Rewrite ILIAS authenticator Joscha 2020-04-20 14:25:55 +00:00
  • ff06c5215e Fix authenticator Joscha 2020-04-20 14:03:22 +00:00
  • 135a8dce4b Fix resolve_path allowing paths outside its folder I-Al-Istannen 2020-04-20 16:07:14 +02:00
  • 63bbcad918 Add resolve method to tmp_dir I-Al-Istannen 2020-04-20 15:39:52 +02:00
  • 6584d6a905 Elaborate accept_file in new_organizer I-Al-Istannen 2020-04-20 15:39:38 +02:00
  • 5990098ef8 Add UserPassAuthenticator Joscha 2020-04-20 13:26:45 +00:00