Completed store results in a database project (epi052#32)

Co-authored-by: Ryan Good <[email protected]>

* added initial skeleton; restructured project directories

* removed workers directive from luigi; changed input to tko-subs

* changed masscan command to use config.tool_paths

* linted __init__ files and updated docstring for get_scans

* added per-file-ignores for linting

* recon-pipeline linted

* PoC working for amass results -> db; rudimentary db mgmt commands also

* more linting

* added database management commands to the shell

* db_location passes through to all tasks; masscan results added to db

* removed unused imports from masscan.py

* added ParseNmapOutput class to handle parsing for database storage

* cleaned up repeat code

* searchsploit results stored in db

* lint/format

* gobuster scans now stored in database

* fixed test_recon tests to use db_location

* fixed web tests

* tkosub entries recorded in db

* subjack scan results stored in database

* webanalyze results stored in db

* refactored older commits to use newer helper functions

* refactored older commits to use newer helper functions

* aquatone results stored in database

refactored a few scans to use dbmanager helper functions
refactored db structure wrt headers/screenshots
added 80/443 to web_ports in config.py

* fixed a few queries and re-added webanalyze to FullScan

* view targets/endpoints done

* overhauled nmap parsing

* print all nmap_results good, next to focus on filtering

* complex nmap filters complete

* nmap printing done

* updated pipfile

* view web-technologies complete

* view searchsploit results complete

* removed filesystem code from amass

* targetlist moved to db only

* targets,amass,masscan all cutover to full database; added view ports

* nmap fully db compliant

* aquatone and webtargets db compliant

* gobuster uses db now

* webanalyze db compliant

* all scans except corscanner are db compliant

* recon tests passing

* web tests passing

* linted files

* added tests for helpers.py and parsers.py

* refactored some redundant code

* added tests to pre-commit

* updated amass tests and pre-commit version

* updated recon.targets tests

* updated nmap tests

* updated masscan tests

* updated config tests

* updated web targets tests

* added gobuster tests

* added aquatone tests

* added subdomain takeover and webanalyze tests; updated test data

* removed homegrown sqlite target in favor of the sqla implementation

* added tests for recon-pipeline.py

* fixed cluge function to set __package__ globally

* updated amass tests

* updated targets tests

* updated nmap tests

* updated masscan tests

* updated aquatone tests

* updated nmap tests to account for no searchsploit

* updated nmap tests to account for no searchsploit

* updated masscan tests

* updated subjack/tkosub tests

* updated web targets tests

* updated webanalyze tests

* added corscanner tests

* linted DBManager a bit

* fixed weird cyclic import issue that only happened during docs build; housekeeping

* added models tests, removed test_install dir

* updated docs a bit; sidenav is wonky

* fixed readthedocs requirements.txt

* fixed issue where view results werent populated directly after scan

* added new tests to pipeline; working on docs

* updated a few overlooked view command items

* updated tests to reflect changes to shell

* incremental push of docs update

* documentation done

* updated exploitdb install

* updated exploitdb install

* updated seclists install

* parseamass updates db in the event of no amass output

* removed corscanner

* added pipenv shell to install instructions per @GreaterGoodest

* added pipenv shell to install instructions per @GreaterGoodest

* added check for chromium-browser during aquatone install; closes epi052#26

* added check for old recon-tools dir; updated Path.resolve calls to Path.expanduser.resolve; fixed very specific import bug due to filesystem location

* added CONTIBUTING.md; updated pre-commit hooks/README

* added .gitattributes for linguist reporting

* updated tests

* fixed a few weird bugs found during test

* updated README

* updated asciinema links in README

* updated README with view command video

* updated other location for url scheme /status

* add ability to specify single target using --target (epi052#31)

* updated a few items in docs and moved tool-dict to tools-dir

* fixed issue where removing tempfile without --verbose caused scan to fail

Author: epi052

.flake8

@@ -2,4 +2,10 @@
 max-line-length = 120
 select = C,E,F,W,B,B950
 ignore = E203, E501, W503
-max-complexity = 13
+max-complexity = 13
+per-file-ignores =
+    pipeline/recon/__init__.py:F401
+    pipeline/models/__init__.py:F401
+    pipeline/recon/web/__init__.py:F401
+    pipeline/luigi_targets/__init__.py:F401
+    tests/test_recon/test_parsers.py:F405

.gitattributes

@@ -0,0 +1 @@
+docs/* linguist-documentation

.github/workflows/pythonapp.yml

@@ -28,14 +28,12 @@ jobs:
       with:
         args: ". --check"
 
-  test-install:
+  test-shell:
 
     runs-on: ubuntu-latest
 
     steps:
     - uses: actions/checkout@v1
-    - name: Set up Golang
-      uses: actions/setup-go@v1
     - name: Set up Python 3.7
       uses: actions/setup-python@v1
       with:
@@ -47,8 +45,8 @@ jobs:
         pipenv install -d
     - name: Test with pytest
       run: |
-        pipenv install pytest cmd2 luigi
-        pipenv run python -m pytest tests/test_install
+        pipenv install pytest cmd2 luigi sqlalchemy python-libnmap
+        pipenv run python -m pytest tests/test_shell
 
   test-recon:
 
@@ -67,7 +65,7 @@ jobs:
         pipenv install -d
     - name: Test with pytest
       run: |
-        pipenv install pytest cmd2 luigi
+        pipenv install pytest cmd2 luigi sqlalchemy python-libnmap
         pipenv run python -m pytest tests/test_recon
 
   test-web:
@@ -87,5 +85,25 @@ jobs:
         pipenv install -d
     - name: Test with pytest
       run: |
-        pipenv install pytest cmd2 luigi
-        pipenv run python -m pytest tests/test_web
+        pipenv install pytest cmd2 luigi sqlalchemy python-libnmap
+        pipenv run python -m pytest tests/test_web
+
+  test-models:
+
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v1
+      - name: Set up Python 3.7
+        uses: actions/setup-python@v1
+        with:
+          python-version: 3.7
+      - name: Set up pipenv
+        run: |
+          python -m pip install --upgrade pip
+          pip install pipenv
+          pipenv install -d
+      - name: Test with pytest
+        run: |
+          pipenv install pytest cmd2 luigi sqlalchemy python-libnmap
+          pipenv run python -m pytest tests/test_models

.pre-commit-config.yaml

@@ -4,9 +4,21 @@ repos:
     hooks:
     - id: black
       language_version: python3.7
-      args: ['.']
--   repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v1.2.3
+      args: ['pipeline', 'tests/test_web', 'tests/test_recon', 'tests/test_shell', 'tests/test_models']
+-   repo: https://gitlab.com/pycqa/flake8
+    rev: 3.7.9
     hooks:
     - id: flake8
-
+-   repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v2.5.0  # Use the ref you want to point at
+    hooks:
+    -   id: trailing-whitespace
+    -   id: debug-statements
+-   repo: local
+    hooks:
+    - id: tests
+      name: run tests
+      entry: pytest
+      language: system
+      types: [python]
+      args: ['tests/test_web', 'tests/test_recon', 'tests/test_shell', 'tests/test_models']