RAIN-27045 Script to generate a GCP resource count and breakdown

resource_management/gcp/README.md

@@ -0,0 +1,74 @@
+# Overview
+
+BASH script to produce a GCP resource breakdown and total resource count.
+It examines every project in the current org. that the executing user has access to (see Roles below).
+If a project request receives permission denied, an error will be displayed on screen, but the remaining projects will still be examined.
+
+# Pre-requisites
+
+A Unix like shell (MacOS/Linux) with gcloud, sed, bc utilities installed is the required execution environment.
+We recommend using the [gcp cloud shell](https://console.cloud.google.com/home/dashboard?cloudshell=true) - it has all the dependencies.
+
+# Roles
+
+If your account has roles/cloudasset.owner or roles/owner on the resource's parent, it has sufficient permissions.
+Please skip to the section API enablement.
+
+Otherwise, the account executing the script must have roles/cloudassset.viewer and roles/serviceusage.serviceUsageConsumer on the parent of the resources to be examined.
+
+### A) We recommend granting at the org level:
+
+`gcloud organizations add-iam-policy-binding TARGET_ORGANIZATION_ID \
+     --member user:USER_ACCOUNT_EMAIL \
+     --role roles/cloudasset.viewer`
+
+`gcloud organizations add-iam-policy-binding TARGET_ORGANIZATION_ID \
+     --member user:USER_ACCOUNT_EMAIL \
+     --role roles/serviceusage.serviceUsageConsumer`
+
+### B) Alternative is granting for each project to be examined:
+
+`gcloud projects add-iam-policy-binding TARGET_PROJECT_ID \
+     --member user:USER_ACCOUNT_EMAIL \
+     --role roles/cloudasset.viewer`
+
+`gcloud projects add-iam-policy-binding TARGET_PROJECT_ID \
+     --member user:USER_ACCOUNT_EMAIL \
+     --role roles/serviceusage.serviceUsageConsumer`
+
+# API enablement
+
+Script requires access to cloudasset API.
+
+### A) We recommend granting for all projects in the org:
+
+1. Download the script cloudasset_enable.sh
+
+wget https://github.com/lacework-dev/scripts/blob/main/resource_management/gcp/cloudasset_enable.sh
+
+2. Run the script:
+
+`chmod +x ./cloudasset_enable.sh; mkdir -p /tmp/lacework; ./cloudasset_enable.sh 2>&1 | tee /tmp/lacework/enable_output`
+
+### B) Alternative is manually granting for each project to be examined:
+
+`gcloud --project <project_id> services enable cloudasset.googleapis.com`
+
+# Usage
+
+1. Download the script:
+
+wget https://github.com/lacework-dev/scripts/blob/main/resource_management/gcp/gcp_asset_breakdown.sh
+
+2. Run the script:
+
+`chmod +x ./gcp_asset_breakdown.sh; mkdir -p /tmp/lacework; ./gcp_asset_breakdown.sh 2>&1 | tee /tmp/lacework/output`
+
+# Results
+
+Summary output is displayed on screen.
+When the script finishes, we recommend uploading the contents of directory:
+
+`/tmp/lacework/`
+
+This can be done in GCP cloud shell by clicking on the more icon (vertical '...') and selecting download.

resource_management/gcp/cloudasset_enable.sh

@@ -0,0 +1,32 @@
+#!/bin/bash
+
+STAR=*
+FILEENDING=".enable"
+
+mkdir -p /tmp/lacework
+gcloud config set accessibility/screen_reader false
+
+var=$(gcloud projects list --filter='lifecycleState:ACTIVE' | sed "1 d" | cut -d ' ' -f 1)
+number_projects=$(echo "$var" | wc -l)
+
+echo "==> Project list:"
+echo $var | tr " " "\n"
+echo "==> Total number of projects = $number_projects"
+
+read -p "Continue to enable on all projects? " -n 1 -r
+echo    # (optional) move to a new line
+if [[ ! $REPLY =~ ^[Yy]$ ]]
+then
+    [[ "$0" = "$BASH_SOURCE" ]] && exit 1 || return 1 # handle exits from shell or function but don't exit interactive shell
+fi
+
+for val in $var; do
+    echo "=> Enabling for Project $val"
+    if gcloud --project $val services enable cloudasset.googleapis.com > /tmp/lacework/$val$FILEENDING
+    then
+       echo "==> Done."
+    else
+       echo "==> Error enabling."
+    fi
+    echo "***************************************"
+done

resource_management/gcp/gcp_asset_breakdown.sh

@@ -0,0 +1,53 @@
+#!/bin/bash
+
+STAR=*
+FILEENDING=".assets"
+
+echo "Installing utility bc"
+sudo apt install -y bc
+
+echo "This script gives a breakdown of resources by asset type for all projects the authenticated account has access to"
+echo "For pre-requisites, including permissions, please see the README"
+echo "It is recommened to use gcp's cloud shell to execute the script"
+echo "Recommended invocation: chmod +x ./gcp_asset_breakdown.sh; mkdir -p /tmp/lacework; ./gcp_asset_breakdown.sh 2>&1 | tee /tmp/lacework/output"
+echo ""
+
+mkdir -p /tmp/lacework
+gcloud config set accessibility/screen_reader false
+
+var=$(gcloud projects list --filter='lifecycleState:ACTIVE' | sed "1 d" | cut -d ' ' -f 1)
+number_projects=$(echo "$var" | wc -l)
+
+echo "==> Project list:"
+echo $var | tr " " "\n"
+echo "==> Total number of projects = $number_projects"
+
+read -p "Continue to summarise assetTypes and count on all projects? " -n 1 -r
+echo    # (optional) move to a new line
+if [[ ! $REPLY =~ ^[Yy]$ ]]
+then
+    [[ "$0" = "$BASH_SOURCE" ]] && exit 1 || return 1 # handle exits from shell or function but don't exit interactive shell
+fi
+
+for val in $var; do
+    echo "=> Examining Project $val"
+    if gcloud asset list --page-size=1000 --project $val | grep "assetType" > /tmp/lacework/$val$FILEENDING
+    then
+       echo "==> Done."
+    else
+       echo "==> Error examining."
+    fi
+    echo "***************************************"
+done
+# Concatenate all assets into a single file, count assets, reduce each line to just the asset count and then sum the counts to find the total.
+cat /tmp/lacework/*.assets > /tmp/lacework/combined; cat /tmp/lacework/combined | sort | uniq -c | sort -bgr > /tmp/lacework/combined_count
+echo "asset-type breakdown"
+cat /tmp/lacework/combined_count
+sed 's/assetType.*$//g' /tmp/lacework/combined_count > /tmp/lacework/combined_count_numbers
+total_assets=$(paste -sd+ /tmp/lacework/combined_count_numbers | bc)
+echo ""
+echo ""
+echo "Total assets=$total_assets"
+echo ""
+echo "Please take a copy of the contents of directory /tmp/lacework and send for more analysis"
+echo "In GCP cloudshell this can be done by clicking on more (the vertical \"...\") and selecting Download"