Skip to content

[GCP] Pin PyOpenSSL version for gcloud service account bug #5068

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

romilbhardwaj
Copy link
Collaborator

gsutil has issues when using a service account. googleapis/google-api-python-client#2554

Repro [needs double checking]:

Deploy API server with GCP enabled and using service account auth:

NAMESPACE=skypilot
kubectl create secret generic gcp-credentials \
  -n $NAMESPACE \
  --from-file=gcp-cred.json=<SERVICE ACCOUNT JSON HERE>

NAMESPACE=skypilot
RELEASE_NAME=skypilot
WEB_USERNAME=skypilot
WEB_PASSWORD=skypilot
AUTH_STRING=$(htpasswd -nb $WEB_USERNAME $WEB_PASSWORD)
helm upgrade --install $RELEASE_NAME skypilot/skypilot-nightly --devel \
  --namespace $NAMESPACE \
  --create-namespace \
  --set ingress.authCredentials=$AUTH_STRING \
  --set apiService.image=berkeleyskypilot/skypilot-nightly:1.0.0.dev20250326 \
  --set gcpCredentials.enabled=true \
  --set gcpCredentials.projectId=<INSERT PROJECT ID HERE> \
  --set-file apiService.config=config.yaml

Define task YAML:

resources:
  cloud: gcp

file_mounts:
  /data:
    name: romil-test
    source: /tmp/mydir

run: |
  ls -l /data

Try sky launch task.yaml. Fails during bucket upload with:

ERROR: module 'OpenSSL.crypto' has no attribute 'sign'

Looks like a known issue: googleapis/google-api-python-client#2554

I fixed by downgrading to pyOpenSSL==24.2.1 on my API server deployment.

Need someone to take over this PR and:

  • Reproduce this reliably. I ran into this on an API server deployment setup from scratch on a GKE cluster but wasn't able to reproduce on my mac.
  • Validate the fix.

@cg505
Copy link
Collaborator

cg505 commented Mar 31, 2025

@zpoint
Copy link
Collaborator

zpoint commented Apr 2, 2025

I need to manually install a specific version of pyopenssl on the Buildkite Docker image to avoid this issue.

@cg505
Copy link
Collaborator

cg505 commented Apr 2, 2025

I kind of think we should just merge this but we should include a constraint >= 23.2.0. Thoughts @aylei @Michaelvll ?

@romilbhardwaj
Copy link
Collaborator Author

Ran into this again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants