remove history

Author: rportilla-databricks

Diff for: .gitignore

@@ -0,0 +1,39 @@
+./aws/base/.terraform/*
+./aws/base/.terraform.lock.hcl
+./aws/base/terraform.tfstate
+./aws/base/terraform.tfstate.backup
+./aws/base_customer_vpc/.terraform/*
+./aws/base_customer_vpc/.terraform.lock.hcl
+./aws/base_customer_vpc/terraform.tfstate
+./aws/base_customer_vpc/terraform.tfstate.backup
+./aws/security/.terraform/*
+./aws/security/.terraform.lock.hcl
+./aws/security/terraform.tfstate
+./aws/security/terraform.tfstate.backup
+./aws/fs_lakehouse/.terraform/*
+./aws/fs_lakehouse/.terraform.lock.hcl
+./aws/fs_lakehouse/terraform.tfstate
+./aws/fs_lakehouse/terraform.tfstate.backup
+./azure/base/.terraform/*
+./azure/base/.terraform.lock.hcl
+./azure/base/terraform.tfstate
+./azure/base/terraform.tfstate.backup
+./azure/managed_vnet/.terraform/*
+./azure/managed_vnet/.terraform.lock.hcl
+./azure/managed_vnet/terraform.tfstate
+./azure/managed_vnet/terraform.tfstate.backup
+./gcp/.terraform/*
+./gcp/.idea/*
+./gcp/terraform.tfstate
+./gcp/terraform.tfstate.backup
+./gcp/.terraform.lock.hcl
+./azure/.idea/*
+./azure/terraform.tfstate
+./azure/terraform.tfstate.backup
+./azure/.terraform.lock.hcl
+
+.idea
+./.idea
+./.idea/*
+/azure/managed_vnet/.terraform/
+/azure/.terraform/

Diff for: LICENSE

@@ -0,0 +1,44 @@
+Databricks FS Lakehouse templates
+
+Copyright (2022) Databricks, Inc.
+
+This library (the "Software") may not be used except in connection with the Licensee's use of the Databricks Platform Services pursuant 
+to an Agreement (defined below) between Licensee (defined below) and Databricks, Inc. ("Databricks"). The Object Code version of the 
+Software shall be deemed part of the Downloadable Services under the Agreement, or if the Agreement does not define Downloadable Services, 
+Subscription Services, or if neither are defined then the term in such Agreement that refers to the applicable Databricks Platform 
+Services (as defined below) shall be substituted herein for “Downloadable Services.”  Licensee's use of the Software must comply at 
+all times with any restrictions applicable to the Downlodable Services and Subscription Services, generally, and must be used in 
+accordance with any applicable documentation. For the avoidance of doubt, the Software constitutes Databricks Confidential Information
+under the Agreement.
+
+Additionally, and notwithstanding anything in the Agreement to the contrary: 
+* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES 
+  OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 
+  LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR 
+  IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+* you may view, make limited copies of, and may compile the Source Code version of the Software into an Object Code version of the
+  Software.  For the avoidance of doubt, you may not make derivative works of Software (or make any any changes to the Source Code 
+  version of the unless you have agreed to separate terms with Databricks permitting such modifications (e.g., a contribution license
+  agreement)).
+
+If you have not agreed to an Agreement or otherwise do not agree to these terms, you may not use the Software or view, copy or compile
+the Source Code of the Software.
+  
+This license terminates automatically upon the termination of the Agreement or Licensee's breach of these terms.  Additionally, 
+Databricks may terminate this license at any time on notice.  Upon termination, you must permanently delete the Software and all
+copies thereof (including the Source Code).
+
+Agreement: the agreement between Databricks and Licensee governing the use of the Databricks Platform Services, which shall be, with
+respect to Databricks, the Databricks Terms of Service located at www.databricks.com/termsofservice, and with respect to Databricks
+Community Edition, the Community Edition Terms of Service located at www.databricks.com/ce-termsofuse, in each case unless Licensee 
+has entered into a separate written agreement with Databricks governing the use of the applicable Databricks Platform Services.
+ 
+Databricks Platform Services: the Databricks services or the Databricks Community Edition services, according to where the Software is used.
+
+Licensee: the user of the Software, or, if the Software is being used on behalf of a company, the company.
+
+Object Code: is version of the Software produced when an interpreter or a compiler translates the Source Code into recognizable and 
+executable machine code.
+
+Source Code: the human readable portion of the Software.
+

Diff for: NOTICE

@@ -0,0 +1,4 @@
+Databricks FS Lakehouse Blueprints
+Copyright 2022 Databricks, Inc.
+
+“This Software includes software developed at Databricks (https://www.databricks.com/) and its use is subject to the included LICENSE file.

Diff for: README.md

@@ -0,0 +1,47 @@
+## Deploy Your Financial Services Lakehouse Architecture 
+
+### Purpose: 
+
+This set of terraform templates is designed to get every FS practitioner and devops team started quickly with financial services best practice setup as well as highly valuable FS-focused libraries directly in your environment. 
+
+<p align="center">
+  <img src="fs_blueprints.jpg" width="700px"/>
+</p>
+
+
+=======
+### Architecture: 
+
+
+### Details on What is Packaged: 
+
+What's include in this Terraform package? 
+
+1. Hardened Cloud Environment (restricted root bucket) for AWS
+2. Basic Example of Permissions using Databricks ACLs and Groups for AWS
+3. Pre-installed Libraries for Creating Common Data Models & Time Series Analytics (AWS | Azure | GCP)
+4. Example Job with Financial Services Quickstarts (AWS | Azure | GCP)
+5. PrivateLink Automation for AWS
+6. Customer-Managed VPC with GCP
+7. NPIP Architecture & Workspace Creation with Azure
+
+
+### AWS 
+
+3 main modules: 
+
+* Workspace from scratch (new)
+* Managed VPC - Private Link workspace
+* Managed VPC - Pre-installed FS libraries, Groups to protect PII, Private Link
+
+
+### Azure 
+
+
+* Workspace from scratch (new)
+* Managed VNET - No public IPs in VNET with private NSGs
+
+
+### GCP 
+
+* Bring-your-own-VPC configuration with GCP

Diff for: aws/base/cross-account-role.tf

@@ -0,0 +1,27 @@
+data "databricks_aws_assume_role_policy" "this" {
+  external_id = var.databricks_account_id
+}
+
+resource "aws_iam_role" "cross_account_role" {
+  name               = "${local.prefix}-crossaccount"
+  assume_role_policy = data.databricks_aws_assume_role_policy.this.json
+  tags               = var.tags
+}
+
+data "databricks_aws_crossaccount_policy" "this" {
+}
+
+resource "aws_iam_role_policy" "this" {
+  name   = "${local.prefix}-policy"
+  role   = aws_iam_role.cross_account_role.id
+  policy = data.databricks_aws_crossaccount_policy.this.json
+}
+
+resource "databricks_mws_credentials" "this" {
+  provider         = databricks.mws
+  account_id       = var.databricks_account_id
+  role_arn         = aws_iam_role.cross_account_role.arn
+  credentials_name = "${local.prefix}-creds"
+  depends_on       = [aws_iam_role_policy.this]
+}
+

Diff for: aws/base/init.tf

@@ -0,0 +1,24 @@
+terraform {
+  required_providers {
+    databricks = {
+      source  = "databrickslabs/databricks"
+    }
+    aws = {
+      source = "hashicorp/aws"
+      version = "3.49.0"
+    }
+  }
+}
+
+provider "aws" {
+  region = var.region
+}
+
+// initialize provider in "MWS" mode to provision new workspace
+provider "databricks" {
+  alias    = "mws"
+  host     = "https://accounts.cloud.databricks.com"
+  username = var.databricks_account_username
+  password = var.databricks_account_password
+}
+

Diff for: aws/base/root-bucket.tf

@@ -0,0 +1,34 @@
+resource "aws_s3_bucket" "root_storage_bucket" {
+  bucket = "${local.prefix}-rootbucket"
+  acl    = "private"
+  versioning {
+    enabled = false
+  }
+  force_destroy = true
+  tags = merge(var.tags, {
+    Name = "${local.prefix}-rootbucket"
+  })
+}
+
+resource "aws_s3_bucket_public_access_block" "root_storage_bucket" {
+  bucket             = aws_s3_bucket.root_storage_bucket.id
+  ignore_public_acls = true
+  depends_on         = [aws_s3_bucket.root_storage_bucket]
+}
+
+data "databricks_aws_bucket_policy" "this" {
+  bucket = aws_s3_bucket.root_storage_bucket.bucket
+}
+
+resource "aws_s3_bucket_policy" "root_bucket_policy" {
+  bucket     = aws_s3_bucket.root_storage_bucket.id
+  policy     = data.databricks_aws_bucket_policy.this.json
+  depends_on = [aws_s3_bucket_public_access_block.root_storage_bucket]
+}
+
+resource "databricks_mws_storage_configurations" "this" {
+  provider                   = databricks.mws
+  account_id                 = var.databricks_account_id
+  bucket_name                = aws_s3_bucket.root_storage_bucket.bucket
+  storage_configuration_name = "${local.prefix}-storage"
+}

Diff for: aws/base/vars.tf

@@ -0,0 +1,37 @@
+variable "databricks_account_username" {}
+variable "databricks_account_password" {}
+variable "databricks_account_id" {
+  type=string
+  description = "env variable"
+}
+
+variable "tags" {
+  default = {}
+}
+
+variable "cidr_block" {
+  default = "10.4.0.0/16"
+}
+
+variable "region" {
+  default = "us-east-1"
+}
+
+locals {
+region_bucket_policy = (
+  replace(var.region, "-", "_")
+  )
+}
+
+
+
+resource "random_string" "naming" {
+  special = false
+  upper   = false
+  length  = 5
+}
+
+locals {
+  prefix = "fsi-ws"
+}
+

Diff for: aws/base/vpc.tf

@@ -0,0 +1,81 @@
+data "aws_availability_zones" "available" {}
+
+module "vpc" {
+  source  = "terraform-aws-modules/vpc/aws"
+  version = "3.2.0"
+
+  name = local.prefix
+  cidr = var.cidr_block
+  azs  = data.aws_availability_zones.available.names
+  tags = var.tags
+
+  enable_dns_hostnames = true
+  enable_nat_gateway   = true
+  single_nat_gateway   = true
+  create_igw           = true
+
+  public_subnets  = [cidrsubnet(var.cidr_block, 3, 0)]
+  private_subnets = [cidrsubnet(var.cidr_block, 3, 1),
+                     cidrsubnet(var.cidr_block, 3, 2)]
+
+  manage_default_security_group = true
+  default_security_group_name = "${local.prefix}-sg"
+
+  default_security_group_egress = [{
+    cidr_blocks = "0.0.0.0/0"
+  }]
+
+  default_security_group_ingress = [{
+    description = "Allow all internal TCP and UDP"
+    self        = true
+  }]
+}
+
+module "vpc_endpoints" {
+  source = "terraform-aws-modules/vpc/aws//modules/vpc-endpoints"
+  version = "3.2.0"
+
+  vpc_id             = module.vpc.vpc_id
+  security_group_ids = [module.vpc.default_security_group_id]
+
+  endpoints = {
+    s3 = {
+      service         = "s3"
+      service_type    = "Gateway"
+      route_table_ids = flatten([
+        module.vpc.private_route_table_ids, 
+        module.vpc.public_route_table_ids])
+      tags            = {
+        Name = "${local.prefix}-s3-vpc-endpoint"
+      }
+    },
+    sts = {
+      service             = "sts"
+      private_dns_enabled = true
+      subnet_ids          = module.vpc.private_subnets
+      tags                = {
+        Name = "${local.prefix}-sts-vpc-endpoint"
+      }
+    },
+    kinesis-streams = {
+      service             = "kinesis-streams"
+      private_dns_enabled = true
+      subnet_ids          = module.vpc.private_subnets
+      tags                = {
+        Name = "${local.prefix}-kinesis-vpc-endpoint"
+      }
+    },    
+  }
+
+  tags = var.tags
+}
+
+resource "databricks_mws_networks" "this" {
+  provider           = databricks.mws
+  account_id         = var.databricks_account_id
+  network_name       = "${local.prefix}-network"
+  security_group_ids = [module.vpc.default_security_group_id]
+  subnet_ids         = module.vpc.private_subnets
+  vpc_id             = module.vpc.vpc_id
+}
+

Diff for: aws/base/workspace.tf

@@ -0,0 +1,49 @@
+resource "databricks_mws_workspaces" "this" {
+  provider        = databricks.mws
+  account_id      = var.databricks_account_id
+  aws_region      = var.region
+  workspace_name  = local.prefix
+  deployment_name = local.prefix
+
+  credentials_id           = databricks_mws_credentials.this.credentials_id
+  storage_configuration_id = databricks_mws_storage_configurations.this.storage_configuration_id
+  network_id               = databricks_mws_networks.this.network_id
+
+  token {
+    comment = "terraform"
+  }
+}
+
+
+output "databricks_host" {
+  value = databricks_mws_workspaces.this.workspace_url
+}
+
+// initialize provider in normal mode
+provider "databricks" {
+  // in normal scenario you won't have to give providers aliases
+  alias = "created_workspace"
+  host = databricks_mws_workspaces.this.workspace_url
+  username = var.databricks_account_username
+  password = var.databricks_account_password
+}
+
+
+// create PAT token to provision entities within workspace
+resource "databricks_token" "pat" {
+  provider = databricks.created_workspace
+  comment  = "Terraform Provisioning"
+  lifetime_seconds = 86400
+}
+
+// export token for integration tests to run on
+output "databricks_token" {
+  value     = databricks_token.pat.token_value
+  sensitive = true
+}
+
+
+// export token for integration tests to run on
+output "databricks_workspace_id" {
+  value     = databricks_mws_networks.this.workspace_id
+}