|
1 | 1 | <p align="center"> |
2 | | - <img src="https://github.com/michenriksen/gitrob/raw/master/static/images/gopher_full.png" alt="Gitrob" width="200" /> |
| 2 | + <img src="./static/images/gopher_full.png" alt="Gitrob" width="200" /> |
3 | 3 | </p> |
4 | | -<br /> |
5 | | -<br /> |
6 | | -<br /> |
7 | 4 |
|
8 | | -# Gitrob: Putting the Open Source in OSINT |
| 5 | +# Wraith: Putting the Open Source in OSINT |
9 | 6 |  |
10 | 7 |
|
11 | 8 |  |
12 | 9 |
|
13 | 10 | [](https://travis-ci.org/mattyjones/gitrob) |
14 | 11 |
|
15 | | -Gitrob is a tool to help find potentially sensitive files pushed to public repositories on Github. Gitrob will clone repositories belonging to a user or organization down to a configurable depth and iterate through the commit history and flag files that match signatures for potentially sensitive files. The findings will be presented through a web interface for easy browsing and analysis. |
16 | 12 |
|
17 | | -## Usage |
| 13 | +Wraith is a tool to help find potentially sensitive information pushed to repositories on GitLab or Github. Wraith will clone repositories belonging to a user or group/organization down to a configurable depth and iterate through the commit history and flag files and/or commit content that match signatures for potentially sensitive information. The findings will be presented through a web interface for easy browsing and analysis. |
18 | 14 |
|
19 | | - gitrob [options] target [target2] ... [targetN] |
| 15 | +## Features |
20 | 16 |
|
21 | | -### Options |
| 17 | +- Scan the following sources: |
| 18 | + - Gitlab repositories |
| 19 | + - Github.com repositories |
| 20 | + - Local git repos |
| 21 | +- Exclude files, paths, and extensions |
| 22 | +- Web interface for real-time results |
| 23 | +- Configurable commit depth |
| 24 | +- Use environment variables, a config file, or flags |
| 25 | +- Uses sub-commands for easier, more modular, functionality |
| 26 | +- Clone a repo to memory instead of disk |
22 | 27 |
|
23 | | -``` |
24 | | --bind-address string |
25 | | - Address to bind web server to (default "127.0.0.1") |
26 | | --commit-depth int |
27 | | - Number of repository commits to process (default 500) |
28 | | --debug |
29 | | - Print debugging information |
30 | | --enterprise-upload-url string |
31 | | - Upload URL for Github Enterprise (defaults to the URL set in -enterprise-url if any) |
32 | | --enterprise-url string |
33 | | - URL for Github Enterprise |
34 | | --enterprise-user string |
35 | | - Username for Github Enterprise (defaults to first target) |
36 | | --github-access-token string |
37 | | - GitHub access token to use for API requests |
38 | | --include-forks |
39 | | - Include forked repositories in scan |
40 | | --load string |
41 | | - Load session file |
42 | | --no-expand-orgs |
43 | | - Don't add members to targets when processing organizations |
44 | | --no-server |
45 | | - Disables web server |
46 | | --port int |
47 | | - Port to run web server on (default 9393) |
48 | | --save string |
49 | | - Save session to file |
50 | | --silent |
51 | | - Suppress all output except for errors |
52 | | --threads int |
53 | | - Number of concurrent threads (default number of logical CPUs) |
54 | | --gather-all |
55 | | - Specify whether to pull all repositories from the domain |
| 28 | +This currently in beta, check the [roadmap][1] for planned functionality |
| 29 | + |
| 30 | +## Usage |
| 31 | + |
| 32 | +For a full list of use cases and configuration options use the included help functionality. |
| 33 | + |
| 34 | +`gitrob --help` |
| 35 | + |
| 36 | + |
| 37 | +## Configuration |
| 38 | + |
| 39 | +**IMPORTANT** If you are targeting a GitLab group, please give the **group ID** as the target argument. You can find the group ID just below the group name in the GitLab UI. Otherwise, names with suffice for the target arguments. This id can be found on the group homepage. |
| 40 | + |
| 41 | +There are multiple was to configure the tool for a scan. The easiest way is via commandline flags. To get a full list of available flags and their purpose use `gitrob <subcommand> --help`. This will pring out a list of flags and how they interact with the base scan. You can also set all flags as environment variables or use a static config file in YAML format. This config file can be used to store targets for multiple scan targets. |
| 42 | + |
| 43 | +The order of precendence with each item taking precedence over the item below it is: |
| 44 | + |
| 45 | +- explicit call to Set |
| 46 | +- commandline flag |
| 47 | +- environment variable |
| 48 | +- configuration file |
| 49 | +- key/value store |
| 50 | +- default value |
| 51 | + |
| 52 | +The various values are configured independently of each other so if you set all values in a config file, you can then override just the ones you want on the commandline. A sample config file looks like: |
| 53 | + |
| 54 | +```yaml |
| 55 | +--- |
| 56 | +commit-depth: 0 |
| 57 | +gitlab-targets: |
| 58 | + - codeemitter |
| 59 | + - mattyjones1 |
| 60 | + - 8692959 |
| 61 | +silent: false |
| 62 | +debug: true |
| 63 | +gitlab-api-token: <token> |
| 64 | +github-api-token: <token> |
| 65 | +github-targets: |
| 66 | + - mattyjones |
| 67 | + - phantomSecrets |
| 68 | +ignore-path: |
| 69 | + - cmd/ |
| 70 | + - docs/ |
| 71 | +ignore-extension: |
| 72 | + - .go |
| 73 | + - .log |
| 74 | +in-mem-clone: true |
| 75 | +repo-dirs: |
| 76 | + - ../../../mattyjones/telegraf |
56 | 77 | ``` |
57 | 78 |
|
58 | | -### Saving session to a file |
| 79 | +## Examples |
59 | 80 |
|
60 | | -By default, gitrob will store its state for an assessment in memory. This means that the results of an assessment is lost when Gitrob is closed. You can save the session to a file by using the `-save` option: |
| 81 | +Scan a GitLab group assuming your access token has been added to the environment variable or a config file. Look for file signature matches only: |
61 | 82 |
|
62 | | - gitrob -save ~/gitrob-session.json acmecorp |
| 83 | + gitrob scanGitlab <gitlab_group_id> |
63 | 84 |
|
64 | | -Gitrob will save all the gathered information to the specified file path as a special JSON document. The file can be loaded again for browsing at another point in time, shared with other analysts or parsed for custom integrations with other tools and systems. |
| 85 | +Scan a multiple GitLab groups assuming your access token has been added to the environment variable or a config file. Clone repositories into memory for faster analysis. Set the scan mode to 2 to scan each file match for a content match before creating a result.: |
65 | 86 |
|
66 | | -### Loading session from a file |
| 87 | + gitrob scanGitlab -in-mem-clone -mode 2 "<gitlab_group_id_1> <gitlab_group_id_2>" |
67 | 88 |
|
68 | | -A session stored in a file can be loaded with the `-load` option: |
| 89 | +Scan a GitLab groups assuming your access token has been added to the environment variable or a config file. Clone repositories into memory for faster analysis. Set the scan mode to 3 to scan each commit for content matches only.: |
69 | 90 |
|
70 | | - gitrob -load ~/gitrob-session.json |
| 91 | + gitrob scanGitlab -in-mem-clone -mode 3 "<gitlab_group_id>" |
71 | 92 |
|
72 | | -Gitrob will start its web interface and serve the results for analysis. |
| 93 | +Scan a Github user setting your Github access token as a parameter. Clone repositories into memory for faster analysis. |
73 | 94 |
|
74 | | -### Use with Github Enterprise |
| 95 | + gitrob scangithub -github-access-token <token> -in-mem-clone "<github_user_name>" |
75 | 96 |
|
76 | | -To configure Gitrob for Github Enterprise, the following switches can be used: |
| 97 | +### Editing File and Content Regular Expressions |
77 | 98 |
|
78 | | -- `enterprise-url`: Must be specified; this is the URL where the path `/api/v3/` exists. This is usually the URL where the Github web interface can be found. Example: `-enterprise-url=https://github.yourcompany.com` |
79 | | -- `enterprise-upload-url:` Optional, defaults to `enterprise-url`; full path to the upload URL if different from the main Github Enterprise URL. Example: `-enterprise-upload-url=https://github.yourcompany.com/api/v3/upload` |
80 | | -- `enterprise-user`: Optional, defaults to the first target. Example: `-enterprise-user=your.username` |
| 99 | +Regular expressions are included in the [filesignatures.json](./rules/filesignatures.json) and [contentsignatures.json](./rules/contentsignatures.json) files respectively. Edit these files to adjust your scope and fine-tune your results. |
| 100 | +
|
| 101 | +Gitrob will start its web interface and serve the results for analysis. |
81 | 102 |
|
82 | 103 | ## Installation |
83 | 104 |
|
84 | | -A [precompiled version is available](https://github.com/michenriksen/gitrob/releases) for each release, alternatively you can use the latest version of the source code from this repository in order to build your own binary. |
| 105 | +At this stage the only option is to build from source from this repository. |
85 | 106 |
|
86 | | -Make sure you have a correctly configured **Go >= 1.11** environment and that `$GOPATH/bin` is in your `$PATH` |
| 107 | +To install from source, make sure you have a correctly configured **Go >= 1.14** environment and that `$GOPATH/bin` is in your `$PATH`. |
87 | 108 |
|
88 | | - $ go get github.com/michenriksen/gitrob |
| 109 | + $ git clone [email protected]:mattyjones1/gitrob.git |
| 110 | + $ cd ~/go/src/gitrob |
| 111 | + $ make build |
| 112 | + $ ./bin/gitrob-<ARCH> <sub-command> |
| 113 | + |
| 114 | +In the future there will be binary releases of the code |
89 | 115 |
|
90 | | -This command will download gitrob, install its dependencies, compile it and move the `gitrob` executable to `$GOPATH/bin`. |
| 116 | +## Access Tokens |
91 | 117 |
|
92 | | -### Github access token |
| 118 | +Gitrob will need either a GitLab or Github access token in order to interact with the appropriate API. You can create a [GitLab personal access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html), or [a Github personal access token](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/) and save it in an environment variable in your `.bashrc` or similar shell configuration file: |
93 | 119 |
|
94 | | -Gitrob will need a Github access token in order to interact with the Github API. [Create a personal access token](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/) and save it in an environment variable in your `.bashrc` or similar shell configuration file: |
| 120 | + export GITROB_GITLAB_ACCESS_TOKEN=deadbeefdeadbeefdeadbeefdeadbeefdeadbeef |
| 121 | + export GITROB_GITHUB_ACCESS_TOKEN=deadbeefdeadbeefdeadbeefdeadbeefdeadbeef |
95 | 122 |
|
96 | | - export GITROB_ACCESS_TOKEN=deadbeefdeadbeefdeadbeefdeadbeefdeadbeef |
| 123 | +Alternatively you can specify the access token with the `-gitlab-access-token` or `-github-access-token` option on the command line, but watch out for your command history! A configuration file can also be used, an example is provided above. |
97 | 124 |
|
98 | | -Alternatively you can specify the access token with the `-github-access-token` option, but watch out for your command history! |
| 125 | +[1]: docs/development/roadmap.md |
0 commit comments