This repository was archived by the owner on Sep 20, 2024. It is now read-only.
Architecture
Depending on the crawler we choose, a different architecture will be required.
- 👍 less coupled components
- 👍 highly efficient + scalable
- 👍 results available as they come in
- 👎 setup of this architecture could be tough
- 👍 just works
- 👎 results available after a job comes in -> requires a notification mechanism