Add alternative to PrivateDnsName on AWS cloud provider

[greenboxal]

/kind feature

AWS cloud provider forces the hostname, or at least the node name, to be the same as AWS' PrivateDnsName, this leads to the node being removed if a different node name is used.

It should be allowed to use any name as the node name and map it to a tag.

Is there any problem with this? I can provide a PR. It's really cumbersome to use ip-x-x-x-x.region.compute.internal as it doesn't pair with the rest of my infrastructure.

Environment:

• Kubernetes version (use kubectl version): 1.7.5
• Cloud provider or hardware configuration: AWS
• OS: Arch Linux
• Kernel (e.g. uname -a): Linux ip-10-0-50-10 4.12.8-1-ec2 Unit test coverage in Kubelet is lousy. (~30%) #1 SMP Mon Aug 21 22:43:04 PDT 2017 x86_64 GNU/Linux
• Install tools: None

[greenboxal]

@kubernetes/sig-aws-feature-requests

[k8s-ci-robot]

@greenboxal: Reiterating the mentions to trigger a notification:
@kubernetes/sig-aws-feature-requests.

In response to this:

@kubernetes/sig-aws-feature-requests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[smarterclayton]

/lifecycle frozen

[smarterclayton]

In general, I hear a substantial amount of real world operational challenges with users who do not want to be using ip-X-X-X-X.ec2.internal names for their nodes. Other cloud providers have better and more consistent naming that reduces human toil and frustration. I think we should look at addressing this. Came up in the Reddit AMA.

[smarterclayton]

/remove-lifecycle rotten

[filiy]

This change would be great to see. Clusters full of useless node names makes management harder than it needs to be. The AWS ec2 name=value tag would be ideal for node name.

If that can't happen another option could be to set user data on the ec2 instance that way it can be queried locally.

[cloud-user@ip-172-16-1-181 ~]$ curl http://169.254.169.254/latest/user-data
name=bastion1
[cloud-user@ip-172-16-1-181 ~]$ 

[gtaylor]

Count us in, re: hostname frustration. We'd like to name our nodes k8s-<cluster-name>-<instanceid> instead of the IP. This makes it easy to quickly determine which cluster a node belongs to when alerts fire, or when we're looking through the AWS console.

Does GCP use labels, or is it a separate flag to kubelet? I thought one of the flags covered this case, but didn't do anything for AWS. A bit fuzzy on the last time I researched the issue.

[nckturner]

Some relevant notes:

For those that are interested in using instance ID, I found this comment stating that:

InstanceID is not yet suitable for use as a Node.Name

but it doesn't go into any reason why, I'm curious to understand the reasoning.

All the code related to mapping the node to node name just forces private DNS name:

// mapInstanceToNodeName maps a EC2 instance to a k8s NodeName, by extracting the PrivateDNSName
func mapInstanceToNodeName(i *ec2.Instance) types.NodeName {
	return types.NodeName(aws.StringValue(i.PrivateDnsName))
}

[greenboxal]

I was investigating this some time ago, and seems like the name needs to be a valid and resolvable DNS name. I don't remember where I saw that. Even so, those are different problems and I'm ok with having this requirement as long as it can be any DNS name that I choose.

[liggitt]

xref #61878

[usernkey]

Did anyone find a solution/workaround to fix this?

I'm having some issue with it too.

Unable to register node "ip-10-1-17-189.eu-west-2.compute.internal" with API server: nodes "ip-10-1-17-189.eu-west-2.compute.internal" is forbidden: node "ip-10-1-17-189" cannot modify node

[cabrinha]

This issue keeps getting sent to stale/rotten but it is ongoing.

There are several other issues and PRs open and closed that bring this up.

What we need is the option to set the nodeName at the time a worker joins the cluster. kubeadm and kubelet have these options available, its just that setting the cloud-provider to aws omits these options.

This still an issue and lots of users want the option to set their nodeName to something other than ip-X-X-X-X.compute.internal

[liggitt]

as discussed in #61878 and #54482, when running with a cloud provider, components not located on the node must be able to build a mapping from node name to the instance in metadata. That means the mapping should not be based solely on a flag or config local to the node. Looking at an alternative attribute in metadata (not modifiable by the node) is likely a reasonable way to do this.

This was attempted in #61878, but stalled out. I don't think there were objections to the approach, just a need for someone to build out the design, think through the implications on existing clusters, make sure the change is aligned with other cloud provider extraction efforts, and shepherd the change to completion.

[discount-elf]

This broken functionality prevents us from utilizing the AWS provider in hand rolled clusters, and also EKS. Honestly the issue could be avoided if the private domain functionality in route53 and the VPC feature correctly populated the private DNS name in EC2.

I guess however in 2019 it's too much to ask for a VPC with custom internal DNS, like you know, the setups I created for myself back in 1999.

[krish7919]

It is frustrating that AWS treats us-east-1 in a special manner and has a node name separate from the host name in the region which breaks the externalTrafficPolicy: Local functionality in a cluster.

[dhanvi]

/sig cloud-provider

[farvour]

yawn...

This broken functionality prevents us from utilizing the AWS provider in hand rolled clusters, and also EKS. Honestly the issue could be avoided if the private domain functionality in route53 and the VPC feature correctly populated the private DNS name in EC2.

I guess however in 2019 it's too much to ask for a VPC with custom internal DNS, like you know, the setups I created for myself back in 1999.

Yep. Kind of surreal and unbelievable that nobody gives a shit about this problem and just takes the ec2.internal domain at face value and shrugging off the root of the issue: piss-poor decision-making on how to "authorize" the node with the cloud provider and then just sticking with that PrivateDnsName for :allthethings: moving forward.

I can't fight this too much as I haven't dug into the internals, but I've seen a lot of solutions offered and just closed/ignored/swatted down because the person who created it either couldn't be bothered to go through Kubernetes'/Googles' CLA. I'm no better by fanning the flame here. Nevertheless, these are the type of things that irritate me about AWS overall. As @smarterclayton pointed out, the DNS inconsistencies in AWS are pretty bad. Only recently have they begun to go "hey maybe this ec2.internal thing ISNT for everyone? Who woulda thunk...."

Little to late guys...

[andrewsykim]

/assign @nckturner
/triage accepted

[TBBle]

I don't believe this will be changed in the in-tree AWS Cloud Provider, as it is legacy code and is scheduled to be removed.

The state of the external cloud provider's behaviour with --hostname-override is described at https://cloud-provider-aws.sigs.k8s.io/prerequisites/, per kubernetes/cloud-provider-aws#63 (comment).

[dims]

in-tree AWS Cloud Provider has been removed, please use the out of tree one and raise issues in that repository.

/close

[k8s-ci-robot]

@dims: Closing this issue.

In response to this:

in-tree AWS Cloud Provider has been removed, please use the out of tree one and raise issues in that repository.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.