enable_irsa=true creates OIDC provider with audience pointing to global sts endpoint (sts.amazonaws.com)

[toytown]

I have issues

I'm submitting a...

• bug report
• [x ] feature request
• support request - read the FAQ first!
• kudos, thank you, warm fuzzy

What is the current behavior?

EKE- 1.18, Fargate

when enable_irsa = true an OIDC provider is created correctly and the audience is set to "sts.amazonaws.com". However, in most corporate network one is encouraged to use local endpoint "sts.eu-central-1.amazonaws.com". In most of the corporate network I have experienced the "sts.amazonaws.com" is blocked which creates issues for role assumption by containers.

If this is a bug, how to reproduce? Please include a code sample if relevant.

What's the expected behavior?

Either based on region setting of the EKS cluster or through explicit setting of sts endpoint it would be nice to have to be able to configure OIDC audience (sts endpoint) explicitly with some property like

oidc_audience = sts.xxxx.com (if irsa is set tot true)

Are you able to fix this problem and submit a PR? Link here if you have already.

Environment details

• Affected module version:
• OS: Linux
• Terraform version: 0.13.4

Any other relevant info

[damacus]

The local endpoints are usually handled by the VPC module, and I think it should probably stick that way to keep the complexity of this module to a minimum.

The fix for this is probably documentation on how to easily create the endpoint with the VPC module in an example.

[bodgit]

I think @toytown is simply asking for a new variable to allow setting the audience, (the client_id_list argument on the aws_iam_openid_connect_provider resource) which is currently hardcoded to local.sts_principal.

I am probably going to want this also as I have to use VPC endpoints in order to reach the STS API without using a proxy.

[cabrinha]

Deploying to CN accounts, in regions like cn-north-1 cause an error with IAM Roles for Service Accounts.

"Incorrect Token Audience" will be seen because the client_id_list gets configured with sts.amazonaws.com.cn and not sts.amazonaws.com.

[daroga0002]

does this is not working in this use case:

terraform-aws-eks/local.tf

Line 47 in 1e2c324

sts_principal = "sts.${data.aws_partition.current.dns_suffix}"

or this is related to OIDM role module used in examples?

[cabrinha]

does this is not working in this use case:

terraform-aws-eks/local.tf

Line 47 in 1e2c324

sts_principal = "sts.${data.aws_partition.current.dns_suffix}"

or this is related to OIDM role module used in examples?

The code you've pasted here works as expected, but cluster-autoscaler, velero, etc -- almost any service that uses IAM Roles for Service Accounts needs to use sts.amazonaws.com and not sts.amazonaws.com.cn.

It would be nice if we had a variable here to add more.

[daroga0002]

ok, but in that case I believe change is applicable to module https://github.com/terraform-aws-modules/terraform-aws-iam/tree/v4.1.0/modules/iam-assumable-role-with-oidc which is used in example:

terraform-aws-eks/examples/irsa/irsa.tf

Lines 1 to 9 in 9022013

	module "iam_assumable_role_admin" {
	source = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
	version = "3.6.0"
	create_role = true
	role_name = "cluster-autoscaler"
	provider_url = replace(module.eks.cluster_oidc_issuer_url, "https://", "")
	role_policy_arns = [aws_iam_policy.cluster_autoscaler.arn]
	oidc_fully_qualified_subjects = ["system:serviceaccount:${local.k8s_service_account_namespace}:${local.k8s_service_account_name}"]
	}

[cabrinha]

ok, but in that case I believe change is applicable to module https://github.com/terraform-aws-modules/terraform-aws-iam/tree/v4.1.0/modules/iam-assumable-role-with-oidc which is used in example:

terraform-aws-eks/examples/irsa/irsa.tf

Lines 1 to 9 in 9022013

	module "iam_assumable_role_admin" {
	source = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
	version = "3.6.0"
	create_role = true
	role_name = "cluster-autoscaler"
	provider_url = replace(module.eks.cluster_oidc_issuer_url, "https://", "")
	role_policy_arns = [aws_iam_policy.cluster_autoscaler.arn]
	oidc_fully_qualified_subjects = ["system:serviceaccount:${local.k8s_service_account_namespace}:${local.k8s_service_account_name}"]
	}

I don't understand where sts.amazonaws.com comes into play here... The EKS module creates the OpenID Connect Identity Provider, and thats where the Audience aka Client ID is specified...

[cabrinha]

This is what I think needs changing: https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/irsa.tf#L12

AWS docs instruct using sts.amazonaws.com when creating an OIDC provider: https://docs.amazonaws.cn/en_us/eks/latest/userguide/enable-iam-roles-for-service-accounts.html

To create an IAM OIDC identity provider for your cluster with the Amazon Web Services Management Console

    Open the Amazon EKS console at https://console.amazonaws.cn/eks/home#/clusters

.

Select the name of your cluster and then select the Configuration tab.

In the Details section, note the value of the OpenID Connect provider URL.

Open the IAM console at https://console.amazonaws.cn/iam/

.

In the navigation panel, choose Identity Providers. If a Provider is listed that matches the URL for your cluster, then you already have a provider for your cluster. If a provider isn't listed that matches the URL for your cluster, then you must create one.

To create a provider, choose Add Provider.

For Provider Type, choose OpenID Connect.

For Provider URL, paste the OIDC issuer URL for your cluster, and then choose Get thumbprint.

For Audience, enter sts.amazonaws.com and choose Add provider.

[daroga0002]

just to confirm (as I dont have any experience with cn region), https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/irsa.tf#L12 should be pointing always to sts.amazonaws.com ?

[cabrinha]

just to confirm (as I dont have any experience with cn region), https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/irsa.tf#L12 should be pointing always to sts.amazonaws.com ?

When using IAM Roles for Service Accounts, this is the only thing that works for me. I don't see any harm in having both sts.amazonaws.com and sts.amazonaws.com.cn in there though, which is what I'm currently doing.

[cabrinha]

@daroga0002 could you take another look or weigh in on the PR?

[daroga0002]

ok, I understand now your pain

I also checked docs:
https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/CA_with_AWS_IAM_OIDC.md

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.