mirrors/pferd
1
0
Fork 0
mirror of https://github.com/Garmelon/PFERD.git synced 2023-12-21 10:23:01 +01:00
Code Issues Packages Projects Releases Wiki Activity
562 Commits 6 Branches 43 Tags 1.6 MiB
8ec3f41251
Commit Graph

562 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
I-Al-Istannen
8891041069 WIP: crawler: Add opencast video crawler 2020-04-21 23:01:19 +02:00
I-Al-Istannen
70d63e3e90 WIP: Start small ILIAS crawler 2020-04-21 13:32:03 +02:00
I-Al-Istannen
b2a7af2e3e Store modification_date in IliasDownloadInfo, remove parameters 2020-04-21 13:31:50 +02:00
I-Al-Istannen
23bed48c8c Satisfy autopep8 2020-04-21 13:30:42 +02:00
Joscha
0926d33798 Use downloader-specific data classes 2020-04-20 18:07:45 +00:00
I-Al-Istannen
55ba2f4070 Fix pylint in downloaders 2020-04-20 19:49:15 +02:00
I-Al-Istannen
d18b48aaf4 Stream in http downloader 2020-04-20 19:45:25 +02:00
Joscha
4ef0ffe3bf Listen to pylint and mypy 2020-04-20 17:44:58 +00:00
Joscha
ce77995c8f Rename http downloader module 2020-04-20 17:08:51 +00:00
I-Al-Istannen
ed9245c14d Remove old organizer 2020-04-20 18:50:23 +02:00
I-Al-Istannen
01e6972c96 Add ilias downloader 2020-04-20 18:49:01 +02:00
I-Al-Istannen
8181ae5b17 Guard http response in context manager 2020-04-20 18:47:46 +02:00
Joscha
6407190ae0 Soupify requests responses properly 2020-04-20 16:38:30 +00:00
I-Al-Istannen
87395faac2 Add base for simple HTTP downloader 2020-04-20 17:43:59 +02:00
I-Al-Istannen
a9e6e7883d Create temp dir folder in constructor 2020-04-20 17:43:59 +02:00
Joscha
154d6b29dd Listen to pylint 2020-04-20 15:16:22 +00:00
I-Al-Istannen
62ac569ec4 Revert "Add proposed crawler entry type" 
This reverts commit 9f1a0a58ab.

Each crawler will have its own data class.
 2020-04-20 16:59:20 +02:00
I-Al-Istannen
9f1a0a58ab Add proposed crawler entry type 2020-04-20 16:54:47 +02:00
Joscha
879a2c7c80 Rewrite ILIAS authenticator 2020-04-20 14:26:30 +00:00
Joscha
ff06c5215e Fix authenticator 2020-04-20 14:26:29 +00:00
I-Al-Istannen
135a8dce4b Fix resolve_path allowing paths outside its folder 
This happened if the directory name was a prefix of the offending file name.
 2020-04-20 16:07:14 +02:00
I-Al-Istannen
63bbcad918 Add resolve method to tmp_dir 2020-04-20 15:40:07 +02:00
I-Al-Istannen
6584d6a905 Elaborate accept_file in new_organizer 2020-04-20 15:40:07 +02:00
Joscha
5990098ef8 Add UserPassAuthenticator 2020-04-20 13:26:45 +00:00
I-Al-Istannen
f3d3d6bb65 Add some docs to cookie_jar 2020-04-20 14:38:03 +02:00
I-Al-Istannen
b2fe7cc064 Add preliminary logging to organizer and tmp_dir 2020-04-20 14:37:44 +02:00
I-Al-Istannen
930d821dd7 Add a simple organizer 2020-04-20 14:29:48 +02:00
I-Al-Istannen
5c2ff14839 Add "prompt_yes_no" to utils 2020-04-20 14:29:48 +02:00
I-Al-Istannen
a3d6dc7873 Clean up temp_folder 2020-04-20 14:29:48 +02:00
Joscha
53ad1c924b Add cookie jar 2020-04-20 11:35:26 +00:00
I-Al-Istannen
8c431c7d81 Add a simple temporary folder 2020-04-20 12:08:52 +02:00
Joscha
d5dd5aac06 Fix some mypy errors 2020-04-20 01:54:47 +00:00
Joscha
7d48972967 Configure mypy 2020-04-19 19:50:17 +00:00
Joscha
25043a4aaa Remove unnecessary files 
Also document some plans for the new program structure in REWRITE.md
 2020-04-19 19:49:43 +00:00
Joscha
7ebeef5873 Clean up gitignore 2020-04-19 18:47:44 +00:00
Garmelon
9b658776ca
Merge pull request #6 from I-Al-Istannen/master 
Hack in support for TI exams
 2020-03-01 23:09:32 +00:00
I-Al-Istannen
cf3553175f Add OS_Exams synchronizer 2020-02-27 14:51:29 +01:00
I-Al-Istannen
bf8b3cf9f7 Hack in support for TI exams 
This just adds an additional crawl check for AlteKlausuren. This is not
present on the root site but at the suffix `/Klausuren`.
Example config:

```py
 # The "Klausur" needs to be copied verbatim!
ti.synchronize("Klausur", "sync dir name",
               transform=ro_19_klausur_transform, filter=ro_19_klausur_filter)
```
 2020-02-24 20:58:27 +01:00
Garmelon
4a5600d5ce
Merge pull request #5 from I-Al-Istannen/master 
Lose 50 minutes of my life (and fix the TGI tut)
 2019-12-12 11:52:05 +00:00
I-Al-Istannen
f5bc49160f Lose 50 minutes of my life (and fix the TGI tut) 2019-12-12 12:50:16 +01:00
Garmelon
fa5f82d312
Merge pull request #4 from I-Al-Istannen/master 
[TGI] Add TGi tut
 2019-11-18 22:12:32 +00:00
I-Al-Istannen
4433696509 [TGI] Add TGi tut 2019-11-18 09:58:16 +01:00
Garmelon
1f5475abc5
Merge pull request #3 from I-Al-Istannen/master 
Download all TGI files and not just lectures
 2019-10-17 21:10:21 +00:00
I-Al-Istannen
1407c6d264 Download all TGI files and not just lectures 2019-10-17 22:14:32 +02:00
Garmelon
e152bfc4a3
Merge pull request #2 from I-Al-Istannen/master 
Add support for TGI
 2019-10-15 20:01:10 +00:00
I-Al-Istannen
1973c931bd Add support for other years in TGI downloader 2019-10-15 15:37:52 +02:00
I-Al-Istannen
458cc1c6d6 Add support for TGI website 2019-10-15 15:34:59 +02:00
Joscha
52852d11a6 Bump version to 1.1.8 2019-09-22 11:56:41 +00:00
Joscha
f94629a7fa Fix exceptions with weird content types 
(hopefully)
 2019-09-22 11:55:47 +00:00
Joscha
c8ee456d33 Bump version to 1.1.7 2019-07-26 08:14:55 +00:00