mirrors/pferd
1
0
Fork 0
mirror of https://github.com/Garmelon/PFERD.git synced 2023-12-21 10:23:01 +01:00
Code Issues Packages Projects Releases Wiki Activity
416 Commits 6 Branches 43 Tags 1.6 MiB
1bbc0b705f
Commit Graph

294 Commits

Author SHA1 Message Date
I-Al-Istannen
b2a2b5999b Implement ILIAS auth and crawl home page 
This commit introduces the necessary machinery to authenticate with
ILIAS and crawl the home page.

It can't do much yet and just silently fetches the homepage.
 2021-05-15 15:25:05 +02:00
Joscha
595de88d96 Fix authenticator and crawler names 
Now, the "auth:" and "crawl:" parts are considered part of the name. This fixes
crawlers not being able to find their authenticators.
 2021-05-15 15:25:05 +02:00
Joscha
a6fdf05ee9 Allow variable whitespace in arrow rules 2021-05-15 15:25:05 +02:00
Joscha
f897d7c2e1 Add name variants for all arrows 2021-05-15 15:25:05 +02:00
Joscha
b0f731bf84 Make crawlers use transformers 2021-05-15 15:25:05 +02:00
Joscha
302b8c0c34 Fix errors loading local crawler config 
Apparently getint and getfloat may return a None even though this is not
mentioned in their type annotations.
 2021-05-15 15:25:05 +02:00
Joscha
acd674f0a0 Change limiter logic 
Now download tasks are a subset of all tasks.
 2021-05-15 15:25:05 +02:00
Joscha
ed2e19a150 Add reasons for invalid values 2021-05-15 15:25:05 +02:00
Joscha
296a169dd3 Make limiter logic more complex 
The limiter can now distinguish between crawl and download actions and has a
fancy slot system and delay logic.
 2021-05-15 15:25:05 +02:00
Joscha
1591cb9197 Add options to slow down local crawler 
These options are meant to make the local crawler behave more like a
network-based crawler for purposes of testing and debugging other parts of the
code base.
 2021-05-15 15:25:01 +02:00
Joscha
0c9167512c Fix output dir 
I missed these while renaming the resolve function. Shame on me for not running
mypy earlier.
 2021-05-14 21:28:38 +02:00
Joscha
a673ab0fae Delete old files 
I should've done this earlier
 2021-05-14 21:27:44 +02:00
Joscha
6e5fdf4e9e Set user agent to "pferd/<version>" 2021-05-14 21:27:44 +02:00
Joscha
93a5a94dab Single-source version number 2021-05-14 21:27:44 +02:00
Joscha
d565df27b3 Add HttpCrawler 2021-05-13 22:28:14 +02:00
Joscha
e3ee4e515d Disable highlighting of primitives 
This commit prevents rich from highlighting python-looking syntax like numbers,
arrays, 'None' etc.
 2021-05-13 19:47:44 +02:00
Joscha
94d6a01cca Use file mtime in local crawler 2021-05-13 19:42:40 +02:00
Joscha
38bb66a776 Update file metadata in more cases 
PFERD now not only updates file metadata when a file is successfully added or
changed, but also when a file is downloaded and then detected to be unchanged.

This could occur for example if a remote file's modification time was bumped,
possibly because somebody touched the file without changing it.
 2021-05-13 19:40:10 +02:00
Joscha
68781a88ab Fix asynchronous methods being not awaited 2021-05-13 19:39:49 +02:00
Joscha
910462bb72 Log stuff happening to files 2021-05-13 19:37:27 +02:00
Joscha
6bd6adb977 Fix tmp file names 2021-05-13 19:36:46 +02:00
Joscha
0acdee15a0 Let crawlers obtain authenticators 2021-05-13 18:57:20 +02:00
Joscha
c3ce6bb31c Fix crawler cleanup not being awaited 2021-05-11 00:28:45 +02:00
Joscha
0459ed093e Add simple authenticator 
... including some required authenticator infrastructure
 2021-05-11 00:28:03 +02:00
Joscha
d5f29f01c5 Use global conductor instance 
The switch from crawler-local conductors to a single pferd-global conductor was
made to prepare for auth section credential providers.
 2021-05-11 00:05:04 +02:00
Joscha
595ba8b7ab Remove dummy crawler 2021-05-10 23:47:46 +02:00
Joscha
cec0a8e1fc Fix mymy errors 2021-05-09 01:45:01 +02:00
Joscha
f9b2fd60e2 Document local crawler and auth 2021-05-09 01:33:47 +02:00
Joscha
60cd9873bc Add local file crawler 2021-05-06 01:02:40 +02:00
Joscha
273d56c39a Properly load crawler config 2021-05-05 23:45:10 +02:00
Joscha
5497dd2827 Add @noncritical and @repeat decorators 2021-05-05 23:36:54 +02:00
Joscha
bbfdadc463 Implement output directory 2021-05-05 18:08:34 +02:00
Joscha
07e831218e Add sync report 2021-05-02 00:56:10 +02:00
Joscha
91c33596da Load crawlers from config file 2021-04-30 16:22:14 +02:00
Joscha
e7a51decb0 Elaborate on transforms and implement changes 2021-04-29 20:24:18 +02:00
Joscha
f776186480 Use PurePath instead of Path 
Path should only be used when we need to access the file system. For all other
purposes (mainly crawling), we use PurePath instead since the paths don't
correspond to paths in the local file system.
 2021-04-29 20:20:25 +02:00
Joscha
0096d83387 Simplify Limiter implementation 2021-04-29 20:20:25 +02:00
Joscha
502654d853 Fix mypy errors 2021-04-29 15:47:52 +02:00
Joscha
d2103d7c44 Document crawler 2021-04-29 15:43:20 +02:00
Joscha
d96a361325 Test and fix exclusive output 2021-04-29 15:27:16 +02:00
Joscha
2e85d26b6b Use conductor via context manager 2021-04-29 14:23:28 +02:00
Joscha
6431a3fb3d Fix some mypy errors 2021-04-29 14:23:09 +02:00
Joscha
ac3bfd7388 Make progress bars easier to use 
The crawler now supports two types of progress bars
 2021-04-29 13:53:16 +02:00
Joscha
3ea86d18a0 Jerry-rig DummyCrawler to run 2021-04-29 13:45:04 +02:00
Joscha
bbc792f9fb Implement Crawler and DummyCrawler 2021-04-29 13:44:29 +02:00
Joscha
7e127cd5cc Clean up and fix conductor and limiter 
Turns out you have to await an async lock, who knew...
 2021-04-29 13:44:04 +02:00
Joscha
c4fb92c658 Make type hints compatible with Python 3.8 2021-04-29 13:11:58 +02:00
Joscha
a18db57e6f Implement terminal conductor 2021-04-29 11:44:47 +02:00
Joscha
b915e393dd Implement limiter 2021-04-29 10:24:28 +02:00
Joscha
3a74c23d09 Implement transformer 2021-04-29 09:51:50 +02:00
Joscha
fbebc46c58 Load and dump config 2021-04-29 09:51:50 +02:00
Joscha
5595a908d8 Configure entry point 2021-04-27 00:32:21 +02:00
I-Al-Istannen
29cd5d1a3c Reflect totality of sanitize_windows_path in return type 2021-04-19 11:10:02 +02:00
I-Al-Istannen
1f2af3a290 Retry on more I/O Errors 2021-04-13 11:43:22 +02:00
I-Al-Istannen
14cdfb6a69 Fix typo in date demangler doc 2021-04-13 11:19:51 +02:00
I-Al-Istannen
946b7a7931 Also crawl .c/.java/.zip from IPD page 2021-02-09 12:30:59 +01:00
I-Al-Istannen
fb78a6e98e Retry ILIAS downloads a few times and only fail that file 2021-01-06 13:08:10 +01:00
I-Al-Istannen
f0562049b6 Remove Python 3.9 method in crawler 2020-12-30 17:18:04 +01:00
I-Al-Istannen
c978e9edf4 Resolve a few pylint warnings 2020-12-30 14:45:46 +01:00
I-Al-Istannen
2714ac6be6 Send CSRF token to Shibboleth 2020-12-30 14:34:11 +01:00
I-Al-Istannen
9b048a9cfc Canonize meeting names to a properly formatted date 2020-12-30 14:32:59 +01:00
I-Al-Istannen
f47b137b59 Fix ILIAS init.py and Pferd.py authenticators 2020-12-06 13:15:32 +01:00
Scriptim
83ea15ee83 Use system keyring service for password auth 2020-12-06 13:15:30 +01:00
I-Al-Istannen
0f5e55648b Tell user when the conflict resolver kept existing files 2020-12-05 14:12:45 +01:00
I-Al-Istannen
4ce385b262 Treat file overwrite and marked file overwrite differently 2020-12-05 14:03:43 +01:00
I-Al-Istannen
fcb3884a8f Add --remote-first, --local-first and --no-delete flags 2020-12-05 13:49:05 +01:00
I-Al-Istannen
9f6dc56a7b Use a strategy to decide conflict resolution 2020-12-02 19:32:57 +01:00
Christophe
f3a4663491 Add passive/no_prompt flag 2020-12-02 18:24:07 +01:00
I-Al-Istannen
ba3c7f85fa Replace "\" in ILIAS paths as well 
I am not sure whether anybody really uses a backslash in their names,
but I guess it can't hurt to do this for windows users.
 2020-11-19 19:37:28 +01:00
I-Al-Istannen
8ebf0eab16 Sort download summary 2020-11-17 21:36:04 +01:00
I-Al-Istannen
cd90a60dee Move "sanitize_windows_path" to PFERD.transform 2020-11-12 20:52:46 +01:00
I-Al-Istannen
55e9e719ad Sanitize "/" in ilias path names 2020-11-12 20:21:24 +01:00
I-Al-Istannen
316b9d7bf4 Prevent too many retries when fetching an ILIAS page 2020-11-04 22:23:56 +01:00
I-Al-Istannen
f830b42a36 Fix duplicate files in download summary 2020-11-04 21:49:35 +01:00
I-Al-Istannen
ef343dec7c Merge organizer download summaries 2020-11-04 15:06:58 +01:00
I-Al-Istannen
0da2fafcd8 Fix links outside tables 2020-11-04 14:46:15 +01:00
I-Al-Istannen
f4abe3197c Add ipd crawler 2020-11-03 21:15:40 +01:00
I-Al-Istannen
38d4f5b4c9 Do not fail only empty courses 2020-11-03 20:09:54 +01:00
I-Al-Istannen
73c3eb0984 Add option to skip videos in sync_url 2020-10-06 17:20:47 +02:00
I-Al-Istannen
c1ccb6c53e Allow crawling videos with sync_url 2020-10-06 10:46:06 +02:00
I-Al-Istannen
51a713fa04 Allow crawling courses or folders with sync_url 
Video folders do not work, if they are passed directly. Their containing
folder must be specified instead.
 2020-09-28 20:00:01 +02:00
I-Al-Istannen
e32a49480b Expose methods to look up course/element names by id / url 2020-09-28 19:16:52 +02:00
I-Al-Istannen
3f0ae729d6 Expand "is course" check to not download magazines or other weird things 2020-09-28 16:43:58 +02:00
I-Al-Istannen
55678d7fee Pass string down to FileCookieJar 
Some python versions just can't handle it *despite the documentation
stating they should*.
 2020-08-12 09:09:14 +02:00
I-Al-Istannen
a57ee8b96b Add timeout to video downloads to work around requests IPv6 bug 2020-08-11 14:40:30 +02:00
Joscha
77a109bb7e Fix ilias shibboleth authenticator 
The shibboleth site got a visual overhaul that slightly changed the classes of a
form we need.
 2020-07-28 19:13:51 +00:00
I-Al-Istannen
a3e1864a26 Allow long paths on windows 
If you start PFERD a few folders deep in your home directory, it is
quite easy to reach the maximum path length limit on Windows (260
chars). This patch opts in to long paths ("\\?\" prefix) which lift that
restriction at the cost of ugly path names.
 2020-07-25 13:44:49 +02:00
I-Al-Istannen
77874b432b Also add personal_desktop to download summary 2020-07-15 22:47:44 +02:00
I-Al-Istannen
5c4c785e60 Fix HTML file downloading 
Previously PFERD thought any HTML file was a "Error, no access" page
when downloading. Now it checks whether ILIAS sends a
content-disposition header, telling the browser to download the file. If
that is the case, it was just a HTML file uploaded to ILIAS. If it has
no header, it is probably an error message.
 2020-07-15 15:12:14 +02:00
I-Al-Istannen
2aed4f6d1f Only query the dir_filter for directories 2020-07-13 13:36:12 +02:00
I-Al-Istannen
34152fbe54 Set mtime and atime to ILIAS dates where possible 2020-07-13 13:29:18 +02:00
I-Al-Istannen
c26c9352f1 Make DownloadSummary private, provide property accessors 2020-06-26 17:30:45 +02:00
I-Al-Istannen
d9ea688145 Use pretty logger for summaries 2020-06-26 17:24:36 +02:00
I-Al-Istannen
e4b1fac045 Satisfy pylint 2020-06-26 15:38:22 +02:00
Joscha
402ae81335 Fix type hints 2020-06-26 13:17:44 +00:00
Daniel Augustin
52f31e2783
Add type hints to DownloadSummary 2020-06-26 13:02:37 +02:00
Daniel Augustin
739522a151
Move download summary into a separate class 2020-06-25 23:07:11 +02:00
Daniel Augustin
6c034209b6
Add deleted files to summary 2020-06-25 22:00:28 +02:00
Daniel Augustin
f6fbd5e4bb
Add download summary 2020-06-25 19:19:34 +02:00
I-Al-Istannen
7024db1f13 Use transient progessbar 
This will ensure no pesky newline ends up in the output, even on
windows.
 2020-06-25 18:03:12 +02:00