|
|
5b929f09a2
|
Move download strategies to downloader
Also fixes an issue where the downloader didn't mark files that were not
downloaded due to the strategy used.
|
2020-04-24 14:27:40 +00:00 |
|
|
|
4d32f863bc
|
Clean up organizer after synchronizing
|
2020-04-24 14:17:23 +00:00 |
|
|
|
4e7333b396
|
Allow specifying paths as strings in Pferd
|
2020-04-24 11:50:40 +00:00 |
|
|
|
4c0e3b493a
|
Use download_modified_or_new as default strategy
|
2020-04-24 13:48:06 +02:00 |
|
|
|
2de079a5d3
|
Add a few Transform combinators
|
2020-04-24 11:35:46 +00:00 |
|
|
|
509e624d47
|
Satisfy pyling. Useful docstrings? Not quite sure.
|
2020-04-23 20:35:59 +02:00 |
|
|
|
980f69b5af
|
Fix organizer marking itself causing an error
|
2020-04-23 20:02:05 +02:00 |
|
|
|
0b00a9c26b
|
Log when starting to synchronize
|
2020-04-23 19:56:37 +02:00 |
|
|
|
1ef85c45e5
|
Switch Transform to PurePath
|
2020-04-23 17:40:43 +00:00 |
|
|
|
5ef5a56e69
|
Extract Location into separate file
|
2020-04-23 17:38:28 +00:00 |
|
|
|
f3f4be2690
|
More free functions
|
2020-04-23 19:21:49 +02:00 |
|
|
|
076b8c5a1f
|
Add download strategies to save bandwith
Only download files that are newer than the local version.
|
2020-04-23 18:29:20 +02:00 |
|
|
|
13bc78c889
|
Display reason for ignoring an element in ilias crawler
|
2020-04-23 13:54:58 +02:00 |
|
|
|
dc964a9d98
|
Remove finished TODOs
|
2020-04-23 13:30:34 +02:00 |
|
|
|
c2b14f3db9
|
ilias crawler: Use direct download link if possible
|
2020-04-23 13:08:12 +02:00 |
|
|
|
4b59a7c375
|
Move around TODOs
|
2020-04-23 10:49:01 +00:00 |
|
|
|
bef210ae77
|
Rename and implement IliasDirectoryFilter
|
2020-04-23 12:35:18 +02:00 |
|
|
|
ea005517cf
|
Only remove folders if they exist in tmpdir
|
2020-04-23 12:09:45 +02:00 |
|
|
|
df0eb84a44
|
Fix TmpDir and Location
TmpDir: Clean up before and after, not just after
Location: Resolve path so that parent check works properly
|
2020-04-23 09:50:32 +00:00 |
|
|
|
2de4255a78
|
Add Pferd class
|
2020-04-23 09:50:32 +00:00 |
|
|
|
3c808879c9
|
Add Transforms and Transformables
|
2020-04-22 18:25:09 +00:00 |
|
|
|
a051e3bcca
|
ilias crawler: Add some unhelpful documentation
|
2020-04-22 17:58:19 +02:00 |
|
|
|
eb7df036df
|
WIP: ilias crawler: Also crawl assignments
|
2020-04-22 14:32:20 +02:00 |
|
|
|
23db59e733
|
WIP: ilias-crawler: Demangle dates
|
2020-04-22 12:58:44 +02:00 |
|
|
|
ac65b06a8e
|
Satisfy pylint a bit
|
2020-04-22 01:37:34 +02:00 |
|
|
|
8891041069
|
WIP: crawler: Add opencast video crawler
|
2020-04-21 23:01:19 +02:00 |
|
|
|
70d63e3e90
|
WIP: Start small ILIAS crawler
|
2020-04-21 13:32:03 +02:00 |
|
|
|
b2a7af2e3e
|
Store modification_date in IliasDownloadInfo, remove parameters
|
2020-04-21 13:31:50 +02:00 |
|
|
|
23bed48c8c
|
Satisfy autopep8
|
2020-04-21 13:30:42 +02:00 |
|
|
|
0926d33798
|
Use downloader-specific data classes
|
2020-04-20 18:07:45 +00:00 |
|
|
|
55ba2f4070
|
Fix pylint in downloaders
|
2020-04-20 19:49:15 +02:00 |
|
|
|
d18b48aaf4
|
Stream in http downloader
|
2020-04-20 19:45:25 +02:00 |
|
|
|
4ef0ffe3bf
|
Listen to pylint and mypy
|
2020-04-20 17:44:58 +00:00 |
|
|
|
ce77995c8f
|
Rename http downloader module
|
2020-04-20 17:08:51 +00:00 |
|
|
|
ed9245c14d
|
Remove old organizer
|
2020-04-20 18:50:23 +02:00 |
|
|
|
01e6972c96
|
Add ilias downloader
|
2020-04-20 18:49:01 +02:00 |
|
|
|
8181ae5b17
|
Guard http response in context manager
|
2020-04-20 18:47:46 +02:00 |
|
|
|
6407190ae0
|
Soupify requests responses properly
|
2020-04-20 16:38:30 +00:00 |
|
|
|
87395faac2
|
Add base for simple HTTP downloader
|
2020-04-20 17:43:59 +02:00 |
|
|
|
a9e6e7883d
|
Create temp dir folder in constructor
|
2020-04-20 17:43:59 +02:00 |
|
|
|
154d6b29dd
|
Listen to pylint
|
2020-04-20 15:16:22 +00:00 |
|
|
|
62ac569ec4
|
Revert "Add proposed crawler entry type"
This reverts commit 9f1a0a58ab.
Each crawler will have its own data class.
|
2020-04-20 16:59:20 +02:00 |
|
|
|
9f1a0a58ab
|
Add proposed crawler entry type
|
2020-04-20 16:54:47 +02:00 |
|
|
|
879a2c7c80
|
Rewrite ILIAS authenticator
|
2020-04-20 14:26:30 +00:00 |
|
|
|
ff06c5215e
|
Fix authenticator
|
2020-04-20 14:26:29 +00:00 |
|
|
|
135a8dce4b
|
Fix resolve_path allowing paths outside its folder
This happened if the directory name was a prefix of the offending file name.
|
2020-04-20 16:07:14 +02:00 |
|
|
|
63bbcad918
|
Add resolve method to tmp_dir
|
2020-04-20 15:40:07 +02:00 |
|
|
|
6584d6a905
|
Elaborate accept_file in new_organizer
|
2020-04-20 15:40:07 +02:00 |
|
|
|
5990098ef8
|
Add UserPassAuthenticator
|
2020-04-20 13:26:45 +00:00 |
|
|
|
f3d3d6bb65
|
Add some docs to cookie_jar
|
2020-04-20 14:38:03 +02:00 |
|
