Commit Graph

46 Commits

Author SHA1 Message Date
bf8b3cf9f7 Hack in support for TI exams
This just adds an additional crawl check for AlteKlausuren. This is not
present on the root site but at the suffix `/Klausuren`.
Example config:

```py
 # The "Klausur" needs to be copied verbatim!
ti.synchronize("Klausur", "sync dir name",
               transform=ro_19_klausur_transform, filter=ro_19_klausur_filter)
```
2020-02-24 20:58:27 +01:00
f5bc49160f Lose 50 minutes of my life (and fix the TGI tut) 2019-12-12 12:50:16 +01:00
4433696509 [TGI] Add TGi tut 2019-11-18 09:58:16 +01:00
1407c6d264 Download all TGI files and not just lectures 2019-10-17 22:14:32 +02:00
1973c931bd Add support for other years in TGI downloader 2019-10-15 15:37:52 +02:00
458cc1c6d6 Add support for TGI website 2019-10-15 15:34:59 +02:00
f94629a7fa Fix exceptions with weird content types
(hopefully)
2019-09-22 11:55:47 +00:00
2752e98621 Fix relative url joining in ti downloader 2019-07-26 10:06:01 +02:00
ea01dc7cb2 Allow even more types of files 2019-07-05 08:48:43 +00:00
77056e6f8d Allow more types of files 2019-07-04 12:16:42 +00:00
d468a45662 Allow wolfram files 2019-06-11 12:42:55 +00:00
67da4e69fa Add colorful log output
Highlight the important operations (new, modified) in different colours.
2019-06-07 13:28:55 +02:00
2016f61bf8 Crawl more of the TI page 2019-05-09 11:04:24 +00:00
c72e92db18 Make Ti downloader authentication more robust 2019-05-06 12:04:01 +00:00
44b4204517 Add basic Ti downloader 2019-05-06 11:54:36 +00:00
d730d0064c Conform to other files' __all__ 2019-04-26 09:45:24 +00:00
ae6cc40fb5 Rename ILIAS crawler to ilias
To be consistent with the other classes' capitalisation of acronyms
2019-04-26 04:29:12 +00:00
0891e7f1bc Fix logging messages not appearing 2019-04-26 03:58:11 +00:00
9693e1d968 Make logging easier 2019-04-25 19:53:13 +00:00
f1ba618378 Remove unnecessary files 2019-04-25 19:18:19 +00:00
dfddc93039 Move norbert from aiohttp to requests
Also fix streaming (when downloading) in the other classes.
2019-04-25 19:15:36 +00:00
f0c42ce8ec Clean up
Use shorter name for responses, like in the requests doc.

Change Organizer's __all__ to be more in line with the other __all__s.
2019-04-25 19:02:48 +00:00
82adeb324f Move ffm stuff from aiohttp to requests 2019-04-25 19:01:53 +00:00
9bae030186 Move ilias stuff from aiohttp to requests 2019-04-25 18:52:48 +00:00
c7a9a42b3d Allow files of type application/msword 2019-04-24 12:34:50 +00:00
5a1bf2188b Switch from tabs to spaces 2019-04-24 12:34:20 +00:00
3019e4255b Replace "/" in file names with "." 2018-12-14 09:27:12 +00:00
616a8d96a2 Sort norbert files while downloading 2018-12-05 11:44:35 +00:00
2d9223b8e6 Add norbert synchronizer 2018-11-29 10:26:58 +00:00
bdc0e8ad03 Remember files correctly for cleanin up 2018-11-28 08:59:07 +00:00
dad33b8c7f Save identically named files under different names 2018-11-27 17:23:32 +00:00
98a2b5db34 Fix tut crawling 2018-11-27 10:28:39 +00:00
c824ae4f6d Add more allowed file types 2018-11-27 10:27:20 +00:00
8b1a34233a Add and use utility functions for changing paths
This fixes a small bug in the example config, where some files were
put in the wrong locations.
2018-11-27 08:52:27 +00:00
a084b05433 Change log message
for better readability
2018-11-26 17:33:27 +00:00
068fe77dcf Clean up minor things
- improve logging messages
- allow more download file formats
- strip file names
2018-11-26 17:00:17 +00:00
34da5d4d19 Sync files from ILIAS 2018-11-26 13:39:06 +00:00
529c4a7dda Don't overwrite files if the contents match 2018-11-26 13:37:01 +00:00
2034c9d426 Add FfM (Fachschaft für Mathematik) synchronizer
This commit moves exceptions and some other things into utils.py and
renames files according to python's file naming guides (kinda).

It also adds a new example config using the new FfM downloader.
2018-11-24 08:27:33 +00:00
5732268084 Clean up
- detect whether authenticating is really necessary when attempting to
download a file
- add a get_website_refid() function
- move often-used goto.php url into constant
- and some comments
2018-11-23 17:45:07 +00:00
2afcd38f1c Rename Ilias-specific stuff 2018-11-23 10:09:03 +00:00
5d5f60e21f Log properly 2018-11-23 10:08:31 +00:00
282d0252eb Add file organizer 2018-11-23 08:56:59 +00:00
4e6912591c Download files to some local file 2018-11-23 08:53:49 +00:00
cf9d43fe84 Fix authenticating bug 2018-11-21 06:59:34 +00:00
95646b0b29 Authenticate with ILIAS and get pages by refid 2018-11-20 05:55:41 +00:00