Skip to content

[k8s] Support exec based auth kubeconfigs on controllers #4379

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
romilbhardwaj opened this issue Nov 17, 2024 · 2 comments · May be fixed by #5453
Open

[k8s] Support exec based auth kubeconfigs on controllers #4379

romilbhardwaj opened this issue Nov 17, 2024 · 2 comments · May be fixed by #5453
Assignees
Labels
help wanted Extra attention is needed k8s Kubernetes related items

Comments

@romilbhardwaj
Copy link
Collaborator

romilbhardwaj commented Nov 17, 2024

GKE and EKS rely on exec based auth in the kubeconfig to authenticate with the k8s API server. This is currently not supported on SkyPilot controllers.

For example, say the user has two GKE clusters using exec based auth, GKE1 and GKE2:

  • If the controller is launched on GKE1, the controller will be able to launch clusters on GKE1, but not on GKE2.
  • If the controller is launched on any other cloud, the controller will not be able to launch clusters on GKE1 OR GKE2.

Here's the reason why we don't support exec based auth:

Using exec-based authentication is problematic when used in conjunction
with kubernetes.remote_identity = LOCAL_CREDENTIAL in ~/.sky/config.yaml.
This is because the exec-based authentication may not have the relevant
dependencies installed on the remote cluster or may have hardcoded paths
that are not available on the remote cluster.

The current suggested workaround is to create a kubeconfig that uses token based auth with a service account (generate_kubeconfig.sh).

This workaround introduces friction for users, and may not be be feasible in environments where users cannot create service accounts.

We should support exec based auth, maybe starting with supporting GKE and EKS. This would require installing relevant dependencies and copying over the cloud credentials.

@romilbhardwaj romilbhardwaj added the k8s Kubernetes related items label Nov 26, 2024
@romilbhardwaj romilbhardwaj added the help wanted Extra attention is needed label Dec 11, 2024
@Michaelvll Michaelvll added the OSS label Dec 19, 2024 — with Linear
@Michaelvll Michaelvll removed the OSS label Dec 19, 2024
@Michaelvll Michaelvll added the OSS label Dec 19, 2024 — with Linear
@Michaelvll Michaelvll removed the OSS label Dec 19, 2024
@weih1121
Copy link
Contributor

in jobs queue will address soon

@kyuds
Copy link
Collaborator

kyuds commented Apr 18, 2025

willing to tackle!

@kyuds kyuds linked a pull request Apr 30, 2025 that will close this issue
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed k8s Kubernetes related items
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants