Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi az for azure #179

Merged
merged 25 commits into from
Aug 24, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions examples/azure/ca/locals.tf
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
locals {
network = {
cidr = data.terraform_remote_state.network.outputs.network_cidr
name = data.terraform_remote_state.network.outputs.network_name
dns = data.terraform_remote_state.network.outputs.dns_zone_id
id = data.terraform_remote_state.network.outputs.network_id
region = data.terraform_remote_state.network.outputs.region
cidr = data.terraform_remote_state.network.outputs.network_cidr
name = data.terraform_remote_state.network.outputs.network_name
dns = data.terraform_remote_state.network.outputs.dns_zone_id
id = data.terraform_remote_state.network.outputs.network_id
region = data.terraform_remote_state.network.outputs.region
locations = join(",", data.terraform_remote_state.network.outputs.locations)

subnet_id = data.terraform_remote_state.network.outputs.subnet_map["private"]
image_id = data.terraform_remote_state.network.outputs.image_id
Expand Down
11 changes: 6 additions & 5 deletions examples/azure/cassandra/locals.tf
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
locals {
network = {
cidr = data.terraform_remote_state.network.outputs.network_cidr
name = data.terraform_remote_state.network.outputs.network_name
dns = data.terraform_remote_state.network.outputs.dns_zone_id
id = data.terraform_remote_state.network.outputs.network_id
region = data.terraform_remote_state.network.outputs.region
cidr = data.terraform_remote_state.network.outputs.network_cidr
name = data.terraform_remote_state.network.outputs.network_name
dns = data.terraform_remote_state.network.outputs.dns_zone_id
id = data.terraform_remote_state.network.outputs.network_id
region = data.terraform_remote_state.network.outputs.region
locations = join(",", data.terraform_remote_state.network.outputs.locations)

subnet_id = data.terraform_remote_state.network.outputs.subnet_map["cassandra"]
image_id = data.terraform_remote_state.network.outputs.image_id
Expand Down
2 changes: 0 additions & 2 deletions examples/azure/kubernetes/example.tfvars
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
# Optional
# kubernetes_cluster_availability_zones = ["1", "2", "3"]

kubernetes_cluster = {
# name = "scalar-kubernetes"
# dns_prefix = "scalar-kubernetes"
Expand Down
11 changes: 6 additions & 5 deletions examples/azure/kubernetes/locals.tf
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
locals {
network = {
name = data.terraform_remote_state.network.outputs.network_name
dns = data.terraform_remote_state.network.outputs.dns_zone_id
id = data.terraform_remote_state.network.outputs.network_id
region = data.terraform_remote_state.network.outputs.region
cidr = data.terraform_remote_state.network.outputs.network_cidr
name = data.terraform_remote_state.network.outputs.network_name
dns = data.terraform_remote_state.network.outputs.dns_zone_id
id = data.terraform_remote_state.network.outputs.network_id
region = data.terraform_remote_state.network.outputs.region
cidr = data.terraform_remote_state.network.outputs.network_cidr
locations = join(",", data.terraform_remote_state.network.outputs.locations)

bastion_ip = data.terraform_remote_state.network.outputs.bastion_ip
user_name = data.terraform_remote_state.network.outputs.user_name
Expand Down
3 changes: 0 additions & 3 deletions examples/azure/kubernetes/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,6 @@ module "kubernetes" {
# Required variables (use network remote state)
network = local.network

# Required kubernetes variable for AZs
kubernetes_cluster_availability_zones = var.kubernetes_cluster_availability_zones

# Optional variables
kubernetes_cluster = var.kubernetes_cluster
kubernetes_default_node_pool = var.kubernetes_default_node_pool
Expand Down
6 changes: 0 additions & 6 deletions examples/azure/kubernetes/vars.tf
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,6 @@ variable "kubernetes_cluster" {
description = "Custom definition kubernetes properties that include name of the cluster, kubernetes version, etc.."
}

variable "kubernetes_cluster_availability_zones" {
type = list(string)
default = []
description = "Select the available zone for the kubernetes cluster or leave empty if the datacenter does not support AZs"
}

variable "kubernetes_default_node_pool" {
type = map
default = {}
Expand Down
11 changes: 6 additions & 5 deletions examples/azure/monitor/locals.tf
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
locals {
network = {
cidr = data.terraform_remote_state.network.outputs.network_cidr
name = data.terraform_remote_state.network.outputs.network_name
dns = data.terraform_remote_state.network.outputs.dns_zone_id
id = data.terraform_remote_state.network.outputs.network_id
region = data.terraform_remote_state.network.outputs.region
cidr = data.terraform_remote_state.network.outputs.network_cidr
name = data.terraform_remote_state.network.outputs.network_name
dns = data.terraform_remote_state.network.outputs.dns_zone_id
id = data.terraform_remote_state.network.outputs.network_id
region = data.terraform_remote_state.network.outputs.region
locations = join(",", data.terraform_remote_state.network.outputs.locations)

subnet_id = data.terraform_remote_state.network.outputs.subnet_map["private"]
image_id = data.terraform_remote_state.network.outputs.image_id
Expand Down
2 changes: 2 additions & 0 deletions examples/azure/network/example.tfvars
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ name = "example-azure" # maximum of 82 characters

region = "japaneast"

# locations = ["1","2","3"]

public_key_path = "./example_key.pub"

private_key_path = "./example_key"
Expand Down
1 change: 1 addition & 0 deletions examples/azure/network/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ module "network" {
# Required Variables
name = var.name
region = var.region
locations = var.locations
public_key_path = var.public_key_path
private_key_path = var.private_key_path
internal_domain = var.internal_domain
Expand Down
4 changes: 4 additions & 0 deletions examples/azure/network/output.tf
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,10 @@ output "region" {
value = module.network.region
}

output "locations" {
value = module.network.locations
}

output "user_name" {
value = module.network.user_name
}
Expand Down
5 changes: 5 additions & 0 deletions examples/azure/network/vars.tf
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@ variable "name" {}

variable "region" {}

variable "locations" {
type = list(string)
default = []
}

variable "public_key_path" {}

variable "private_key_path" {}
Expand Down
11 changes: 6 additions & 5 deletions examples/azure/scalardl/locals.tf
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
locals {
network = {
cidr = data.terraform_remote_state.network.outputs.network_cidr
name = data.terraform_remote_state.network.outputs.network_name
dns = data.terraform_remote_state.network.outputs.dns_zone_id
id = data.terraform_remote_state.network.outputs.network_id
region = data.terraform_remote_state.network.outputs.region
cidr = data.terraform_remote_state.network.outputs.network_cidr
name = data.terraform_remote_state.network.outputs.network_name
dns = data.terraform_remote_state.network.outputs.dns_zone_id
id = data.terraform_remote_state.network.outputs.network_id
region = data.terraform_remote_state.network.outputs.region
locations = join(",", data.terraform_remote_state.network.outputs.locations)

image_id = data.terraform_remote_state.network.outputs.image_id

Expand Down
1 change: 1 addition & 0 deletions modules/azure/ca/locals.tf
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ locals {
network_dns = var.network.dns
network_id = var.network.id
region = var.network.region
locations = compact(split(",", var.network.locations))
subnet_id = var.network.subnet_id
image_id = var.network.image_id
bastion_ip = var.network.bastion_ip
Expand Down
3 changes: 2 additions & 1 deletion modules/azure/ca/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,13 @@ resource "null_resource" "wait_for" {
}

module "ca_cluster" {
source = "github.com/scalar-labs/terraform-azurerm-compute?ref=c122120"
source = "github.com/scalar-labs/terraform-azurerm-compute?ref=f934b9d"

nb_instances = local.ca.resource_count
admin_username = local.user_name
resource_group_name = local.network_name
location = local.region
availability_zones = local.locations
vm_hostname = "ca"
nb_public_ip = "0"
vm_os_simple = local.image_id
Expand Down
6 changes: 5 additions & 1 deletion modules/azure/cassandra/cassandra.tf
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,13 @@ resource "null_resource" "wait_for" {
}

module "cassandra_cluster" {
source = "github.com/scalar-labs/terraform-azurerm-compute?ref=c122120"
source = "github.com/scalar-labs/terraform-azurerm-compute?ref=f934b9d"

nb_instances = local.cassandra.resource_count
admin_username = local.user_name
resource_group_name = local.network_name
location = local.region
availability_zones = local.locations
vm_hostname = "cassandra"
nb_public_ip = "0"
vm_os_simple = local.image_id
Expand All @@ -27,6 +28,7 @@ resource "azurerm_managed_disk" "cassandra_data_volume" {

name = "data-disk-cassandra${count.index}"
location = local.region
zones = length(local.locations) > 0 ? [element(local.locations, count.index)] : null
resource_group_name = local.network_name
storage_account_type = local.cassandra.data_remote_volume_type
create_option = "Empty"
Expand Down Expand Up @@ -92,6 +94,7 @@ resource "azurerm_managed_disk" "cassandra_commitlog_volume" {

name = "commitlog-cassandra${count.index}"
location = local.region
zones = length(local.locations) > 0 ? [element(local.locations, count.index)] : null
resource_group_name = local.network_name
storage_account_type = local.cassandra.commitlog_remote_volume_type
create_option = "Empty"
Expand Down Expand Up @@ -167,6 +170,7 @@ module "cassandra_provision" {
cassy_public_key = module.cassy_provision.public_key
start_on_initial_boot = local.cassandra.start_on_initial_boot
internal_domain = local.internal_domain
locations = length(local.locations) > 0 ? local.locations : [""]
}

resource "azurerm_private_dns_a_record" "cassandra_dns" {
Expand Down
3 changes: 2 additions & 1 deletion modules/azure/cassandra/cassy.tf
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
module "cassy_cluster" {
source = "github.com/scalar-labs/terraform-azurerm-compute?ref=c122120"
source = "github.com/scalar-labs/terraform-azurerm-compute?ref=f934b9d"

nb_instances = local.cassy.resource_count
admin_username = local.user_name
resource_group_name = local.network_name
location = local.region
availability_zones = local.locations
vm_hostname = "cassy"
nb_public_ip = "0"
vm_os_simple = local.image_id
Expand Down
1 change: 1 addition & 0 deletions modules/azure/cassandra/locals.tf
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ locals {
network_dns = var.network.dns
network_id = var.network.id
region = var.network.region
locations = compact(split(",", var.network.locations))
subnet_id = var.network.subnet_id
image_id = var.network.image_id
bastion_ip = var.network.bastion_ip
Expand Down
3 changes: 2 additions & 1 deletion modules/azure/cassandra/reaper.tf
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
module "reaper_cluster" {
source = "github.com/scalar-labs/terraform-azurerm-compute?ref=c122120"
source = "github.com/scalar-labs/terraform-azurerm-compute?ref=f934b9d"

nb_instances = local.reaper.resource_count
admin_username = local.user_name
resource_group_name = local.network_name
location = local.region
availability_zones = local.locations
vm_hostname = "reaper"
nb_public_ip = "0"
vm_os_simple = local.image_id
Expand Down
1 change: 0 additions & 1 deletion modules/azure/kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ The Azure Kubernetes Module creates a subnet for k8s, service principal, set per
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| kubernetes_scalar_apps_pool | Custom definition kubernetes scalar apps node pools, same as default_node_pool | `map` | `{}` | no |
| kubernetes_cluster_availability_zones | Select the available zone for the kubernetes cluster or leave empty if the datacenter does not support AZs | `list(string)` | `[]` | no |
| kubernetes_cluster | Custom definition kubernetes properties that include name of the cluster, kubernetes version, etc.. | `map` | `{}` | no |
| kubernetes_default_node_pool | Custom definition kubernetes default node pool that include number of node, node size, autoscaling, etc.. | `map` | `{}` | no |
| network | Custom definition for network and bastion | `map` | `{}` | no |
Expand Down
4 changes: 2 additions & 2 deletions modules/azure/kubernetes/kubernetes.tf
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ resource "azurerm_kubernetes_cluster" "aks_cluster" {
name = substr(local.kubernetes_default_node_pool.name, 0, 12)
node_count = local.kubernetes_default_node_pool.node_count
vm_size = local.kubernetes_default_node_pool.vm_size
availability_zones = var.kubernetes_cluster_availability_zones
availability_zones = local.locations
max_pods = local.kubernetes_default_node_pool.max_pods
os_disk_size_gb = local.kubernetes_default_node_pool.os_disk_size_gb
vnet_subnet_id = azurerm_subnet.k8s_private["k8s_node_pod"].id
Expand Down Expand Up @@ -159,7 +159,7 @@ resource "azurerm_kubernetes_cluster_node_pool" "aks_cluster_scalar_apps_node_po
name = substr(local.kubernetes_scalar_apps_pool.name, 0, 12)
node_count = local.kubernetes_scalar_apps_pool.node_count
vm_size = local.kubernetes_scalar_apps_pool.vm_size
availability_zones = var.kubernetes_cluster_availability_zones
availability_zones = local.locations
max_pods = local.kubernetes_scalar_apps_pool.max_pods
os_disk_size_gb = local.kubernetes_scalar_apps_pool.os_disk_size_gb
os_type = local.kubernetes_scalar_apps_pool.node_os
Expand Down
1 change: 1 addition & 0 deletions modules/azure/kubernetes/locals.tf
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ locals {
network_cidr = var.network.cidr
network_id = var.network.id
public_key_path = var.network.public_key_path
locations = compact(split(",", var.network.locations))
}

# Network subnet
Expand Down
6 changes: 0 additions & 6 deletions modules/azure/kubernetes/vars.tf
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,6 @@ variable "kubernetes_cluster" {
description = "Custom definition kubernetes properties that include name of the cluster, kubernetes version, etc.."
}

variable "kubernetes_cluster_availability_zones" {
type = list(string)
default = []
description = "Select the available zone for the kubernetes cluster or leave empty if the datacenter does not support AZs"
}

variable "kubernetes_default_node_pool" {
type = map
default = {}
Expand Down
1 change: 1 addition & 0 deletions modules/azure/monitor/locals.tf
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ locals {
network_dns = var.network.dns
network_id = var.network.id
region = var.network.region
locations = compact(split(",", var.network.locations))
subnet_id = var.network.subnet_id
image_id = var.network.image_id
bastion_ip = var.network.bastion_ip
Expand Down
4 changes: 3 additions & 1 deletion modules/azure/monitor/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,13 @@ resource "null_resource" "wait_for" {
}

module "monitor_cluster" {
source = "github.com/scalar-labs/terraform-azurerm-compute?ref=c122120"
source = "github.com/scalar-labs/terraform-azurerm-compute?ref=f934b9d"

nb_instances = local.monitor.resource_count
admin_username = local.user_name
resource_group_name = local.network_name
location = local.region
availability_zones = local.locations
vm_hostname = "monitor"
nb_public_ip = local.monitor.set_public_access ? 1 : 0
public_ip_dns = ["monitor-${local.network_name}"]
Expand All @@ -29,6 +30,7 @@ resource "azurerm_managed_disk" "monitor_log_volume" {

name = "log-${count.index + 1}"
location = local.region
zones = length(local.locations) > 0 ? [element(local.locations, count.index)] : null
resource_group_name = local.network_name
storage_account_type = local.monitor.log_volume_type
create_option = "Empty"
Expand Down
1 change: 1 addition & 0 deletions modules/azure/network/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ The Network module creates a virtual network with subnets.
| image_id | The image id to initiate. |
| internal_domain | The internal domain for setting srv record. |
| region | The Azure region to deploy environment. |
| locations | The Azure availability zones to deploy environment |
| network_cidr | The virtual network CIDR address space. |
| network_id | The virtual network ID. |
| network_name | Short name to identify environment. |
Expand Down
1 change: 1 addition & 0 deletions modules/azure/network/bastion/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ The bastion module deploys a network bastion host used to access and configure t
| enable_tdagent | A flag to install td-agent that forwards logs to the monitor host | `any` | n/a | yes |
| image_id | The image id to initiate | `any` | n/a | yes |
| region | The Azure region to deploy environment | `any` | n/a | yes |
| locations | The Azure availability zones to deploy environment | `list(string)` | `[]` | no |
| network_cidr | The network cidr address space | `any` | n/a | yes |
| network_dns | The ID for the internal DNS zone | `any` | n/a | yes |
| network_id | The id of the cloud provider network ID resource | `any` | n/a | yes |
Expand Down
10 changes: 5 additions & 5 deletions modules/azure/network/bastion/main.tf
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
module "bastion_cluster" {
# TODO: Fix ref=xxxxxx
source = "github.com/scalar-labs/terraform-azurerm-compute?ref=c122120"
source = "github.com/scalar-labs/terraform-azurerm-compute?ref=f934b9d"

nb_instances = var.resource_count
admin_username = var.user_name
resource_group_name = var.network_name
location = var.region
availability_zones = var.locations
vm_hostname = "bastion"
vm_os_simple = var.image_id
vnet_subnet_id = var.subnet_id
nb_public_ip = "1"
public_ip_dns = ["bastion-${var.network_name}"]
nb_public_ip = var.resource_count
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tei-k I try to start 3 bastions and got the following error, we should add a random to the dns for public IP

Error: Error Creating/Updating Public IP "bastion-2-publicIP" (Resource Group "paul-k8s-azure-s9rt0ek"): network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="DnsRecordInUse" Message="DNS record bastion-paul-k8s-azure-s9rt0ek.westus2.cloudapp.azure.com is already used by another public IP." Details=[]

  on .terraform/modules/network.bastion.bastion_cluster/main.tf line 251, in resource "azurerm_public_ip" "vm":
 251: resource "azurerm_public_ip" "vm" {



Error: Error Creating/Updating Public IP "bastion-3-publicIP" (Resource Group "paul-k8s-azure-s9rt0ek"): network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="DnsRecordInUse" Message="DNS record bastion-paul-k8s-azure-s9rt0ek.westus2.cloudapp.azure.com is already used by another public IP." Details=[]

  on .terraform/modules/network.bastion.bastion_cluster/main.tf line 251, in resource "azurerm_public_ip" "vm":
 251: resource "azurerm_public_ip" "vm" {

Copy link
Contributor Author

@tei-k tei-k Aug 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@penpyt

Excellent point 👍 👍 Fixed in d02cc64.

e.g. Create different public dns for each bastion.

bastion-1-tei-multi-kclkyfk.japaneast.cloudapp.azure.com
bastion-2-tei-multi-kclkyfk.japaneast.cloudapp.azure.com
bastion-3-tei-multi-kclkyfk.japaneast.cloudapp.azure.com

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, Look good to me !

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tei-k one last thing, should we modify the outputs.tf (network modules) as well to include the x number of bastion servers to give the user the public dns in output terraform command.

I say that because Kubernetes must use only one bastion but the tools (helm, kubectl) need to be available on everyone on them.

I propose to keep bastion_ip and create bastions_ip as follow, let's me know what do you think?

output "bastion_ip" {
  value = module.network.bastion_ip
}

output "bastions_ip" {
  [OMIT]
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@penpyt

I think it's OK to add it to the output, but also need to fix aws as well, So let me do it in other pr. (may not need it now.)

public_ip_dns = formatlist("bastion-%s-${var.network_name}", range(1, var.resource_count + 1))
vm_size = var.resource_type
delete_os_disk_on_termination = true
ssh_key = var.public_key_path
Expand All @@ -25,7 +25,7 @@ module "bastion_provision" {
user_name = var.user_name
private_key_path = var.private_key_path
additional_public_keys_path = var.additional_public_keys_path
provision_count = "1"
provision_count = var.resource_count
enable_tdagent = var.enable_tdagent
internal_domain = var.network_dns
}
Expand Down
5 changes: 5 additions & 0 deletions modules/azure/network/bastion/vars.tf
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,11 @@ variable "region" {
description = "The Azure region to deploy environment"
}

variable "locations" {
type = list(string)
description = "The Azure availability zones to deploy environment"
}

variable "public_key_path" {
description = "The path to the public key for SSH access"
}
Expand Down
2 changes: 2 additions & 0 deletions modules/azure/network/locals.tf
Original file line number Diff line number Diff line change
Expand Up @@ -26,4 +26,6 @@ locals {
scalardl_blue = cidrsubnet(local.network.cidr, 8, 3)
scalardl_green = cidrsubnet(local.network.cidr, 8, 4)
}

locations = compact(var.locations)
}
Loading