Skip to content

[k8s] Support Exec-based Auth #5453

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

Conversation

kyuds
Copy link
Collaborator

@kyuds kyuds commented Apr 30, 2025

Fixes #4379

Need more testing and a lot of feedback please. The main hurdle with supporting exec-based auth is that when using exec-based auth we cannot use incluster configurations at all (anything that uses incluster auth without a proper RBAC role will result in a lot of logs saying that localhost:8080 is inaccessible).

Currently tested on:

  1. Start jobs controller on GKE and submit job to the same cluster (this is where the incluster issues come into play)
  2. Start jobs controller on AWS EC2 and submit job to GKE cluster

Need to test:

  1. Whether this works on EKS (main problem will be from having the correct PATH env var set. The gke auth plugin had a similar issue and as a temporary workaround used a sudo symlink, but this could be better. Being able to set a proper PATH env var will be a more general and workable solution)

Tested (run the relevant ones):

  • Code formatting: install pre-commit (auto-check on commit) or bash format.sh
  • Any manual or new tests for this PR (please specify below)
  • All smoke tests: /smoke-test (CI) or pytest tests/test_smoke.py (local)
  • Relevant individual tests: /smoke-test -k test_name (CI) or pytest tests/test_smoke.py::test_name (local)
  • Backward compatibility: /quicktest-core (CI) or pytest tests/smoke_tests/test_backward_compat.py (local)

@kyuds kyuds marked this pull request as draft April 30, 2025 04:42
@Michaelvll Michaelvll requested a review from SeungjinYang April 30, 2025 16:27
@cg505
Copy link
Collaborator

cg505 commented Apr 30, 2025

We may need some hard-coded whitelist of supported exec providers - some kubernetes users may be rolling their own exec-based auth that won't work from the remote VM. Though maybe it's fine for it to just fail in that case.

@SeungjinYang SeungjinYang requested a review from romilbhardwaj May 1, 2025 01:35
@kyuds
Copy link
Collaborator Author

kyuds commented May 2, 2025

We may need some hard-coded whitelist of supported exec providers - some kubernetes users may be rolling their own exec-based auth that won't work from the remote VM. Though maybe it's fine for it to just fail in that case.

I think we should fail, but alert users of supported exec providers through sky check. For exec-based auth to work, say for GKE, users need to enable both GCP AND K8S. This requirement is already logged on sky check, so we could also provide a list of available exec methods there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[k8s] Support exec based auth kubeconfigs on controllers
2 participants