Commit Graph

43 Commits

Author SHA1 Message Date
1407c6d264 Download all TGI files and not just lectures 2019-10-17 22:14:32 +02:00
1973c931bd Add support for other years in TGI downloader 2019-10-15 15:37:52 +02:00
458cc1c6d6 Add support for TGI website 2019-10-15 15:34:59 +02:00
f94629a7fa Fix exceptions with weird content types
(hopefully)
2019-09-22 11:55:47 +00:00
2752e98621 Fix relative url joining in ti downloader 2019-07-26 10:06:01 +02:00
ea01dc7cb2 Allow even more types of files 2019-07-05 08:48:43 +00:00
77056e6f8d Allow more types of files 2019-07-04 12:16:42 +00:00
d468a45662 Allow wolfram files 2019-06-11 12:42:55 +00:00
67da4e69fa Add colorful log output
Highlight the important operations (new, modified) in different colours.
2019-06-07 13:28:55 +02:00
2016f61bf8 Crawl more of the TI page 2019-05-09 11:04:24 +00:00
c72e92db18 Make Ti downloader authentication more robust 2019-05-06 12:04:01 +00:00
44b4204517 Add basic Ti downloader 2019-05-06 11:54:36 +00:00
d730d0064c Conform to other files' __all__ 2019-04-26 09:45:24 +00:00
ae6cc40fb5 Rename ILIAS crawler to ilias
To be consistent with the other classes' capitalisation of acronyms
2019-04-26 04:29:12 +00:00
0891e7f1bc Fix logging messages not appearing 2019-04-26 03:58:11 +00:00
9693e1d968 Make logging easier 2019-04-25 19:53:13 +00:00
f1ba618378 Remove unnecessary files 2019-04-25 19:18:19 +00:00
dfddc93039 Move norbert from aiohttp to requests
Also fix streaming (when downloading) in the other classes.
2019-04-25 19:15:36 +00:00
f0c42ce8ec Clean up
Use shorter name for responses, like in the requests doc.

Change Organizer's __all__ to be more in line with the other __all__s.
2019-04-25 19:02:48 +00:00
82adeb324f Move ffm stuff from aiohttp to requests 2019-04-25 19:01:53 +00:00
9bae030186 Move ilias stuff from aiohttp to requests 2019-04-25 18:52:48 +00:00
c7a9a42b3d Allow files of type application/msword 2019-04-24 12:34:50 +00:00
5a1bf2188b Switch from tabs to spaces 2019-04-24 12:34:20 +00:00
3019e4255b Replace "/" in file names with "." 2018-12-14 09:27:12 +00:00
616a8d96a2 Sort norbert files while downloading 2018-12-05 11:44:35 +00:00
2d9223b8e6 Add norbert synchronizer 2018-11-29 10:26:58 +00:00
bdc0e8ad03 Remember files correctly for cleanin up 2018-11-28 08:59:07 +00:00
dad33b8c7f Save identically named files under different names 2018-11-27 17:23:32 +00:00
98a2b5db34 Fix tut crawling 2018-11-27 10:28:39 +00:00
c824ae4f6d Add more allowed file types 2018-11-27 10:27:20 +00:00
8b1a34233a Add and use utility functions for changing paths
This fixes a small bug in the example config, where some files were
put in the wrong locations.
2018-11-27 08:52:27 +00:00
a084b05433 Change log message
for better readability
2018-11-26 17:33:27 +00:00
068fe77dcf Clean up minor things
- improve logging messages
- allow more download file formats
- strip file names
2018-11-26 17:00:17 +00:00
34da5d4d19 Sync files from ILIAS 2018-11-26 13:39:06 +00:00
529c4a7dda Don't overwrite files if the contents match 2018-11-26 13:37:01 +00:00
2034c9d426 Add FfM (Fachschaft für Mathematik) synchronizer
This commit moves exceptions and some other things into utils.py and
renames files according to python's file naming guides (kinda).

It also adds a new example config using the new FfM downloader.
2018-11-24 08:27:33 +00:00
5732268084 Clean up
- detect whether authenticating is really necessary when attempting to
download a file
- add a get_website_refid() function
- move often-used goto.php url into constant
- and some comments
2018-11-23 17:45:07 +00:00
2afcd38f1c Rename Ilias-specific stuff 2018-11-23 10:09:03 +00:00
5d5f60e21f Log properly 2018-11-23 10:08:31 +00:00
282d0252eb Add file organizer 2018-11-23 08:56:59 +00:00
4e6912591c Download files to some local file 2018-11-23 08:53:49 +00:00
cf9d43fe84 Fix authenticating bug 2018-11-21 06:59:34 +00:00
95646b0b29 Authenticate with ILIAS and get pages by refid 2018-11-20 05:55:41 +00:00