File tree 2 files changed +17
-9
lines changed
2 files changed +17
-9
lines changed Original file line number Diff line number Diff line change 1
1
# Go Webcrawler
2
2
3
3
[ ![ Go] ( https://github.com/Acollie/Go-Webcrawler/actions/workflows/go.yml/badge.svg )] ( https://github.com/Acollie/Go-Webcrawler/actions/workflows/go.yml )
4
- ## Description
5
4
6
- ## Testing
5
+ ## Overview
7
6
8
- ### Unit Tests
7
+ ![ Overview of architecture ] ( /assets/overview.png " Overview ")
9
8
10
- ``` bash go test ./... ```
9
+ ### SQS Queue
10
+
11
+ The SQS queue is used to store the URLs that need to be crawled.
12
+
13
+ ### DynamoDB
14
+
15
+ DynamoDB is used to store the URLs that have been crawled and the URLs that have been found on the page.
11
16
12
- ### Integration Tests
17
+ ### Neptune / Neo4k
18
+
19
+ Neptune is used to store the relationships between the URLs that have been found on the page.
20
+
21
+ ### Tests
22
+
23
+ The tests are written in Go and can be run using the following command:
24
+ ``` bash go test ./... ```
13
25
14
- 1 . ``` docker-compose up -d ```
15
- 2 . ``` aws --endpoint-url=http://localhost:4570 sqs create-queue --queue-name test-queue ``` use the results from this to
16
- add to the ` .env `
17
- 3 . ``` bash go test ./... -tags=integration ```
You can’t perform that action at this time.
0 commit comments