fix: Invalid for_each argument with var.create_cluster_primary_security_group_tags = false

[dex4er]

I've got today:

╷          
│ Error: Invalid for_each argument
│ 
│   on .terraform/modules/app_cluster.eks/main.tf line 70, in resource "aws_ec2_tag" "cluster_primary_security_group":
│   70:   for_each = { for k, v in merge(var.tags, var.cluster_tags, data.aws_default_tags.current.tags) :
│   71:     k => v if local.create && k != "Name" && var.create_cluster_primary_security_group_tags
│   72:   }
│     ├────────────────
│     │ data.aws_default_tags.current.tags is a map of string, known only after apply
│     │ local.create is true
│     │ var.cluster_tags is empty map of string
│     │ var.create_cluster_primary_security_group_tags is false
│     │ var.tags is map of string with 4 elements
│ 
│ The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot
│ determine the full set of keys that will identify the instances of this resource.
│ 
│ When working with unknown values in for_each, it's better to define the map keys statically in your configuration and place
│ apply-time results only in the map values.
│ 
│ Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends on, and
│ then apply a second time to fully converge.
╵

but I have var.create_cluster_primary_security_group_tags = false

It worked with this patch.

[bryantbiggs]

do you have a reproduction configuration that demonstrates your error message?

[dex4er]

@bryantbiggs Unfortunately I don't have it: I have very complex EKS setup which was stopped in the middle of destroying then after I rerun terraform apply I've got this error.

Maybe this is just a quirk in Terraform. Maybe it always break if data is deleted then it can't calculate correctly for_each loop again. I would avoid to use any data in any for_each and in this case simple ternary operator helped.

BTW, I've got the situation when no any new plan was sucessful: I couldn't refresh this data.aws_default_tags.current anymore. I couldn't apply all resources or targeted resource anymore.

[dex4er]

@bryantbiggs
Now I'm convinced a74e980 broke it. Currently I've got the same Error: Invalid for_each argument on another project after irrelevant change.

[bryantbiggs]

unfortunately, without a reproduction I'm not really able to provide any guidance/support

[dex4er]

I'm really disappointed that you don't want to accept trivial patch without long tests that will take a lot of time and $$ on AWS, when the error message is very clear:

The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of keys that will identify the instances of this resource.

It is because in for_each there is data source used and it is just a bad practice and it is documented: https://www.terraform.io/language/meta-arguments/for_each#limitations-on-values-used-in-for_each

The keys of the map (or all the values in the case of a set of strings) must be known values, or you will get an error message that for_each has dependencies that cannot be determined before apply, and a -target may be needed.

For the record:

Everything is fine with v18.29.1 and I started to get any problems with v18.30.0. I use Terraform 1.3.2.

[jkotiuk]

I've run into the same issue today,

We are running version 18.20.5 and wanted to upgrade to latest which is 18.30.2. It has failed due the same error.
Latest version that does not give any errors is 18.29.1:

│ Error: Invalid for_each argument
│ 
│   on .terraform/modules/cluster.eks/main.tf line 71, in resource "aws_ec2_tag" "cluster_primary_security_group":
│   71:   for_each = { for k, v in merge(var.tags, var.cluster_tags, data.aws_default_tags.current.tags) :
│   72:     k => v if local.create && k != "Name" && var.create_cluster_primary_security_group_tags
│   73:   }
│     ├────────────────
│     │ data.aws_default_tags.current.tags is a map of string, known only after apply
│     │ local.create is true
│     │ var.cluster_tags is empty map of string
│     │ var.create_cluster_primary_security_group_tags is true
│     │ var.tags is map of string with 1 element

For reference our eks module config, I've redacted some of the fields but this should give you general idea which ones we are using.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "18.29.1"

  cluster_name    = local.name
  cluster_version = var.eks_version

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  cluster_endpoint_public_access  = false
  cluster_endpoint_private_access = true
  cluster_enabled_log_types       = var.cluster_enabled_log_types

  create_iam_role = false
  iam_role_arn    = var.cluster_iam_role_arn

  #Keep backward compatilility with module version before 18
  prefix_separator                   = ""
  cluster_security_group_name        = local.name
  cluster_security_group_description = "EKS cluster security group."

  cluster_security_group_additional_rules = {
    [redacted]
  }

  node_security_group_additional_rules = {
    [redacted]
  }

  node_security_group_tags = {
    "mytag" = "myvalue"
  }

  create_aws_auth_configmap = true
  manage_aws_auth_configmap = true
  aws_auth_node_iam_role_arns_non_windows = [
    [redacted]
  ]

  aws_auth_roles = [
    [redacted]
  ]

  self_managed_node_group_defaults = {
    [redacted]
  }

  self_managed_node_groups = {
    group1 = {
      [redacted]

      security_group_rules = {
        [redacted]
      }

      block_device_mappings = {
        [redacted]
      }
      tags = {
        "Name" = "myname"
      }
    }
  }

  cluster_encryption_config = [
    [redacted]
  ]

  tags = merge(
    data.aws_default_tags.current.tags,
    {
      Terraform = "true"
    }
  )

  depends_on = [redacted]
}

[bryantbiggs]

@jkotiuk change this

  tags = merge(
    data.aws_default_tags.current.tags,
    {
      Terraform = "true"
    }
  )

To

  tags = {
      Terraform = "true"
    }

[jkotiuk]

That didn't help, I've tried that already

[bryantbiggs]

and what does your provider block look like with the default tags - the reproduction provided does not show any errors

[bryantbiggs]

FYI - we are removing all usage of aws_default_tags in v19. It simply does not work trying to bake it in and instead this needs to be fixed at the AWS provider level ed0c336

[jkotiuk]

In that case I'll wait for v19 with further upgrades.

[github-actions]

This PR has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this PR will be closed in 10 days

[github-actions]

This PR was automatically closed because of stale in 10 days

[antonbabenko]

This issue has been resolved in version 19.0.0 🎉

[dex4er]

This issue has been resolved in version 19.0.0 🎉

I want to thank you for almost (cluster_id -> cluster_name) smooth migration from 18.x to 19.x. Perfect job! Using this module is a pleasure.

[bryantbiggs]

That is very kind, thank you

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.