"Instance market options" with the launch template

[refucktor]

Description

EKS Managed Group should support the "Instance market options" with the launch template.

Versions

• Terraform:

Terraform v1.1.4                                                                                   
on linux_amd64

• Provider(s):

+ provider registry.terraform.io/hashicorp/aws v3.73.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.2.0
+ provider registry.terraform.io/hashicorp/helm v2.4.1
+ provider registry.terraform.io/hashicorp/kubernetes v2.7.1
+ provider registry.terraform.io/hashicorp/tls v3.1.0

• Module:

terraform-aws-modules/eks/aws: 18.2.1 

Reproduction

Steps to reproduce the behavior:

1. Configure this module using the examples
2. Add an "instance_market_options" under the "Managed Node Groups"
3. Plan and apply

Code Snippet to Reproduce

For simplicity I referenced some values by using variables, local or other modules, because the root of the error is from the Launch Template creation

module "dev_eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "18.2.1"

  # cluster configuration
  cluster_name              = local.cluster_name
  cluster_version           = local.cluster_version
  cluster_enabled_log_types = ["audit", "api", "authenticator"]
  cluster_addons = {
    coredns = {
      resolve_conflicts = "OVERWRITE"
    }
    vpc-cni = {
      resolve_conflicts = "OVERWRITE"
    }
  }
  cluster_encryption_config = [{
    provider_key_arn = var.aws_kms_key_eks_arn
    resources        = ["secrets"]
  }]

  # iam config
  iam_role_name            = local.cluster_iam_role_name
  iam_role_path            = local.cluster_iam_path
  iam_role_use_name_prefix = false
  enable_irsa              = true

  # network
  vpc_id     = var.aws_vpc_id
  subnet_ids = var.vpc_private_subnets_ids

  # security groups
  cluster_security_group_name = format("%s-sg", local.cluster_name)
  node_security_group_name    = format("%s-node-sg", local.cluster_name)

  # node groups
  eks_managed_node_group_defaults = {
    disk_size = 50
    platform  = "bottlerocket"
    ami_id    = data.aws_ami.bottlerocket_ami.id
  }
 eks_managed_node_groups = {
    spot_worker = {
      name          = "spot-worker"
      instance_type = "t3a.xlarge"
      min_size      = 0
      max_size      = 3
      desired_size  = 1
      capacity_type = "SPOT"
      taints        = []
      labels = {
        lifecycle = "spot"
      }
      instance_market_options = {
        market_type = "spot"
        spot_options = {
          max_price          = "0.075"
          spot_instance_type = "one-time"
        }
      }
    }
  }

  tags = {
    Name        = local.cluster_name
    Environment = var.environment
  }
}

Expected behavior

If I attempt to create the launch template from the AWS console I can set those values. I would like to have same experience using this module

Actual behavior

Receive this error while applying

 Error: error creating EKS Node Group (dev-cluster:spot-worker-20220204091449689900000020): InvalidParameterException: Instance market options are not supported with the launch template.
│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "fa9876e0-ae73-41ae-95c9-60910f26a99b"
│   },
│   Message_: "Instance market options are not supported with the launch template."
│ }

Terminal Output Screenshot(s)

Additional Content

Here you can see the creation using the AWS Console directly

[bryantbiggs]

The module provides support for instance_market_options

terraform-aws-eks/modules/eks-managed-node-group/main.tf

Line 160 in 2f01a36

dynamic "instance_market_options" {

but AWS EKS does not natively support it https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html#managed-node-group-capacity-types which is why the API error is shown when you try to provision it

[refucktor]

@bryantbiggs thanks for the fast reply! and thanks for your work in this module!

I also noticed even if there is an input for the instance_market_options it cannot be used otherwise we get that error. I also tracked the error to a similar context of the link you share. Considering that this module is internally using the resource "aws_eks_node_group" it makes sense to receive the error.

Workaround

• Just for anybody having this issue, if you want to stick with "Managed Node Groups", a possible workaround could be: Create the Launch Template separately and then reference in the node group definition of the module.
• Switch to the "self_managed_node_groups" property, it should not have that limitation (pending test)

Recommendations:

• There is an input variable for the submodule "eks-managed-node-groups" working as a flag: "create_launch_template" which by default is true, I would recommend that whenever you are using that flag, then re-use the values to set the "instance_market_options". I just tested to create an "aws_eks_node_group" resource using a lunch template with that options and it worked perfectly.

[refucktor]

@bryantbiggs maybe you can help me to understand where is the difference in this case

I created by myself the aws_launch_template and the aws_eks_node_group resources like the following

resource "aws_launch_template" "sample_template" {
  name          = "sample-template"
  instance_type = "t3a.xlarge"
  image_id      = "ami-0451386b18edb6f88"
  vpc_security_group_ids = local.node_group_security_group_ids

  instance_market_options {
    market_type = "spot"
    spot_options {
      max_price          = "0.071"
      spot_instance_type = "one-time"
    }
  }
}

resource "aws_eks_node_group" "sample_group" {
  node_group_name = "sample-group"
  cluster_name    = local.cluster_name
  node_role_arn   = aws_iam_role.this[0].arn
  subnet_ids      = var.vpc_private_subnets_ids

  capacity_type = "SPOT"

  scaling_config {
    desired_size = 1
    max_size     = 3
    min_size     = 0
  }


  lifecycle {
    create_before_destroy = true
    ignore_changes = [
      scaling_config[0].desired_size,
    ]
  }

  launch_template {
    id      = aws_launch_template.sample_template.id
    version = aws_launch_template.sample_template.latest_version
  }
}

I could create both with no problems, "in theory" the logic of the submodule "eks_managed_node_group" is pretty similar because by default there is a launch_template created.
I would really appreciate any help!

[bryantbiggs]

Give me a few to spin up a cluster and check it out further

[bryantbiggs]

so it looks like there might be a loophole in the config you provided - not sure if its because of the custom AMI or what, but market options are currently not supported by the EKS managed node group service:

• Instance market options are not supported with the launch template EKS terraform hashicorp/terraform-provider-aws#15118
• https://docs.aws.amazon.com/eks/latest/APIReference/API_LaunchTemplateSpecification.html

The launch template cannot include SubnetId, IamInstanceProfile, RequestSpotInstances, HibernationOptions, or TerminateInstances, or the node group deployment or update will fail.

I did fine one small bug but thats irrelevant for your issue here (will have a PR shortly - block_duration_minutes is not a required attribute under spot_options)

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

[github-actions]

This issue was automatically closed because of stale in 10 days

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.