12 Commits

Author	SHA1	Message	Date
Joscha	273d56c39a	Properly load crawler config	2021-05-05 23:45:10 +02:00
Joscha	5497dd2827	Add @noncritical and @repeat decorators	2021-05-05 23:36:54 +02:00
Joscha	bbfdadc463	Implement output directory	2021-05-05 18:08:34 +02:00
Joscha	91c33596da	Load crawlers from config file	2021-04-30 16:22:14 +02:00
Joscha	f776186480	Use PurePath instead of Path ... Path should only be used when we need to access the file system. For all other purposes (mainly crawling), we use PurePath instead since the paths don't correspond to paths in the local file system.	2021-04-29 20:20:25 +02:00
Joscha	502654d853	Fix mypy errors	2021-04-29 15:47:52 +02:00
Joscha	d2103d7c44	Document crawler	2021-04-29 15:43:20 +02:00
Joscha	d96a361325	Test and fix exclusive output	2021-04-29 15:27:16 +02:00
Joscha	2e85d26b6b	Use conductor via context manager	2021-04-29 14:23:28 +02:00
Joscha	6431a3fb3d	Fix some mypy errors	2021-04-29 14:23:09 +02:00
Joscha	ac3bfd7388	Make progress bars easier to use ... The crawler now supports two types of progress bars	2021-04-29 13:53:16 +02:00
Joscha	bbc792f9fb	Implement Crawler and DummyCrawler	2021-04-29 13:44:29 +02:00