418 Commits

Author	SHA1	Message	Date
Joscha	6407190ae0	Soupify requests responses properly	2020-04-20 16:38:30 +00:00
I-Al-Istannen	87395faac2	Add base for simple HTTP downloader	2020-04-20 17:43:59 +02:00
I-Al-Istannen	a9e6e7883d	Create temp dir folder in constructor	2020-04-20 17:43:59 +02:00
Joscha	154d6b29dd	Listen to pylint	2020-04-20 15:16:22 +00:00
I-Al-Istannen	62ac569ec4	Revert "Add proposed crawler entry type" ... This reverts commit 9f1a0a58ab. Each crawler will have its own data class.	2020-04-20 16:59:20 +02:00
I-Al-Istannen	9f1a0a58ab	Add proposed crawler entry type	2020-04-20 16:54:47 +02:00
Joscha	879a2c7c80	Rewrite ILIAS authenticator	2020-04-20 14:26:30 +00:00
Joscha	ff06c5215e	Fix authenticator	2020-04-20 14:26:29 +00:00
I-Al-Istannen	135a8dce4b	Fix resolve_path allowing paths outside its folder ... This happened if the directory name was a prefix of the offending file name.	2020-04-20 16:07:14 +02:00
I-Al-Istannen	63bbcad918	Add resolve method to tmp_dir	2020-04-20 15:40:07 +02:00
I-Al-Istannen	6584d6a905	Elaborate accept_file in new_organizer	2020-04-20 15:40:07 +02:00
Joscha	5990098ef8	Add UserPassAuthenticator	2020-04-20 13:26:45 +00:00
I-Al-Istannen	f3d3d6bb65	Add some docs to cookie_jar	2020-04-20 14:38:03 +02:00
I-Al-Istannen	b2fe7cc064	Add preliminary logging to organizer and tmp_dir	2020-04-20 14:37:44 +02:00
I-Al-Istannen	930d821dd7	Add a simple organizer	2020-04-20 14:29:48 +02:00
I-Al-Istannen	5c2ff14839	Add "prompt_yes_no" to utils	2020-04-20 14:29:48 +02:00
I-Al-Istannen	a3d6dc7873	Clean up temp_folder	2020-04-20 14:29:48 +02:00
Joscha	53ad1c924b	Add cookie jar	2020-04-20 11:35:26 +00:00
I-Al-Istannen	8c431c7d81	Add a simple temporary folder	2020-04-20 12:08:52 +02:00
Joscha	d5dd5aac06	Fix some mypy errors	2020-04-20 01:54:47 +00:00
Joscha	25043a4aaa	Remove unnecessary files ... Also document some plans for the new program structure in REWRITE.md	2020-04-19 19:49:43 +00:00
I-Al-Istannen	cf3553175f	Add OS_Exams synchronizer	2020-02-27 14:51:29 +01:00
I-Al-Istannen	bf8b3cf9f7	Hack in support for TI exams ... This just adds an additional crawl check for AlteKlausuren. This is not present on the root site but at the suffix `/Klausuren`. Example config: ```py # The "Klausur" needs to be copied verbatim! ti.synchronize("Klausur", "sync dir name", transform=ro_19_klausur_transform, filter=ro_19_klausur_filter) ```	2020-02-24 20:58:27 +01:00
I-Al-Istannen	f5bc49160f	Lose 50 minutes of my life (and fix the TGI tut)	2019-12-12 12:50:16 +01:00
I-Al-Istannen	4433696509	[TGI] Add TGi tut	2019-11-18 09:58:16 +01:00
I-Al-Istannen	1407c6d264	Download all TGI files and not just lectures	2019-10-17 22:14:32 +02:00
I-Al-Istannen	1973c931bd	Add support for other years in TGI downloader	2019-10-15 15:37:52 +02:00
I-Al-Istannen	458cc1c6d6	Add support for TGI website	2019-10-15 15:34:59 +02:00
Joscha	f94629a7fa	Fix exceptions with weird content types ... (hopefully)	2019-09-22 11:55:47 +00:00
I-Al-Istannen	2752e98621	Fix relative url joining in ti downloader	2019-07-26 10:06:01 +02:00
Joscha	ea01dc7cb2	Allow even more types of files	2019-07-05 08:48:43 +00:00
Joscha	77056e6f8d	Allow more types of files	2019-07-04 12:16:42 +00:00
Joscha	d468a45662	Allow wolfram files	2019-06-11 12:42:55 +00:00
I-Al-Istannen	67da4e69fa	Add colorful log output ... Highlight the important operations (new, modified) in different colours.	2019-06-07 13:28:55 +02:00
Joscha	2016f61bf8	Crawl more of the TI page	2019-05-09 11:04:24 +00:00
Joscha	c72e92db18	Make Ti downloader authentication more robust	2019-05-06 12:04:01 +00:00
Joscha	44b4204517	Add basic Ti downloader	2019-05-06 11:54:36 +00:00
Joscha	d730d0064c	Conform to other files' __all__	2019-04-26 09:45:24 +00:00
Joscha	ae6cc40fb5	Rename ILIAS crawler to ilias ... To be consistent with the other classes' capitalisation of acronyms	2019-04-26 04:29:12 +00:00
Joscha	0891e7f1bc	Fix logging messages not appearing	2019-04-26 03:58:11 +00:00
Joscha	9693e1d968	Make logging easier	2019-04-25 19:53:13 +00:00
Joscha	f1ba618378	Remove unnecessary files	2019-04-25 19:18:19 +00:00
Joscha	dfddc93039	Move norbert from aiohttp to requests ... Also fix streaming (when downloading) in the other classes.	2019-04-25 19:15:36 +00:00
Joscha	f0c42ce8ec	Clean up ... Use shorter name for responses, like in the requests doc. Change Organizer's __all__ to be more in line with the other __all__s.	2019-04-25 19:02:48 +00:00
Joscha	82adeb324f	Move ffm stuff from aiohttp to requests	2019-04-25 19:01:53 +00:00
Joscha	9bae030186	Move ilias stuff from aiohttp to requests	2019-04-25 18:52:48 +00:00
Joscha	c7a9a42b3d	Allow files of type application/msword	2019-04-24 12:34:50 +00:00
Joscha	5a1bf2188b	Switch from tabs to spaces	2019-04-24 12:34:20 +00:00
Joscha	3019e4255b	Replace "/" in file names with "."	2018-12-14 09:27:12 +00:00
Joscha	616a8d96a2	Sort norbert files while downloading	2018-12-05 11:44:35 +00:00
Joscha	2d9223b8e6	Add norbert synchronizer	2018-11-29 10:26:58 +00:00
Joscha	bdc0e8ad03	Remember files correctly for cleanin up	2018-11-28 08:59:07 +00:00
Joscha	dad33b8c7f	Save identically named files under different names	2018-11-27 17:23:32 +00:00
Joscha	98a2b5db34	Fix tut crawling	2018-11-27 10:28:39 +00:00
Joscha	c824ae4f6d	Add more allowed file types	2018-11-27 10:27:20 +00:00
Joscha	8b1a34233a	Add and use utility functions for changing paths ... This fixes a small bug in the example config, where some files were put in the wrong locations.	2018-11-27 08:52:27 +00:00
Joscha	a084b05433	Change log message ... for better readability	2018-11-26 17:33:27 +00:00
Joscha	068fe77dcf	Clean up minor things ... - improve logging messages - allow more download file formats - strip file names	2018-11-26 17:00:17 +00:00
Joscha	34da5d4d19	Sync files from ILIAS	2018-11-26 13:39:06 +00:00
Joscha	529c4a7dda	Don't overwrite files if the contents match	2018-11-26 13:37:01 +00:00
Joscha	2034c9d426	Add FfM (Fachschaft für Mathematik) synchronizer ... This commit moves exceptions and some other things into utils.py and renames files according to python's file naming guides (kinda). It also adds a new example config using the new FfM downloader.	2018-11-24 08:27:33 +00:00
Joscha	5732268084	Clean up ... - detect whether authenticating is really necessary when attempting to download a file - add a get_website_refid() function - move often-used goto.php url into constant - and some comments	2018-11-23 17:45:07 +00:00
Joscha	2afcd38f1c	Rename Ilias-specific stuff	2018-11-23 10:09:03 +00:00
Joscha	5d5f60e21f	Log properly	2018-11-23 10:08:31 +00:00
Joscha	282d0252eb	Add file organizer	2018-11-23 08:56:59 +00:00
Joscha	4e6912591c	Download files to some local file	2018-11-23 08:53:49 +00:00
Joscha	cf9d43fe84	Fix authenticating bug	2018-11-21 06:59:34 +00:00
Joscha	95646b0b29	Authenticate with ILIAS and get pages by refid	2018-11-20 05:55:41 +00:00