Skip to content

Latest commit

 

History

History
24 lines (14 loc) · 761 Bytes

README.md

File metadata and controls

24 lines (14 loc) · 761 Bytes

Go Webcrawler 🕷️

Go

Example of a graph

Overview 🌐

Overview of architecture

SQS Queue 📥

The SQS queue is used to store the URLs that need to be crawled.

DynamoDB 📦

DynamoDB is used to store the URLs that have been crawled and the URLs that have been found on the page.

Neptune / Neo4k 🌌

Neptune is used to store the relationships between the URLs that have been found on the page.

Tests

The tests are written in Go and can be run using the following command: bash go test ./...