Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] add serve model with no huggingface request #3245

Closed
devops724 opened this issue Mar 12, 2025 · 5 comments
Closed

[Feature] add serve model with no huggingface request #3245

devops724 opened this issue Mar 12, 2025 · 5 comments

Comments

@devops724
Copy link

Motivation

on run serve
CUDA_VISIBLE_DEVICES=0,1 lmdeploy serve api_server $1 --model-name image-processor --backend turbomind --server-port 7002 --model-format $2 --tp 2 --log-level INFO --session-len $3

i randomly get this error
File "/home/user/miniconda3/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/OpenGVLab/InternVL2_5-26B-MPO-AWQ/revision/main (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/user/miniconda3/bin/lmdeploy", line 8, in
sys.exit(run())
^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/lmdeploy/cli/entrypoint.py", line 31, in run
args.model_path = get_model(args.model_path, download_dir=download_dir, revision=revision)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/lmdeploy/utils.py", line 210, in get_model
model_path = snapshot_download(pretrained_model_name_or_path, ignore_patterns=['*.pth'], **download_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/_snapshot_download.py", line 155, in snapshot_download
repo_info = api.repo_info(repo_id=repo_id, repo_type=repo_type, revision=revision, token=token)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 2756, in repo_info
return method(
^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 2540, in model_info
r = get_session().get(path, headers=headers, timeout=timeout, params=params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/utils/_http.py", line 93, in send
return super().send(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/requests/adapters.py", line 698, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/OpenGVLab/InternVL2_5-26B-MPO-AWQ/revision/main (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))"), '(Request ID: 31669d3e-6a2a-41e4-a68c-6ea611c3307a)')

Related resources

there should be a parameter model for working offline server in local network
prevent sending request to huggingface if model already downloaded

Additional context

No response

@lvhan028
Copy link
Collaborator

The logs indicate LMDeploy is trying to download model from huggingface hub.
Please check the value of $1, i.e., the model path, can be accessed and loaded successfully by transformers.

@devops724
Copy link
Author

devops724 commented Mar 12, 2025 via email

@lvhan028
Copy link
Collaborator

It INDEED access the local model path if the model exists and could be loaded by transformers

@lvhan028
Copy link
Collaborator

lvhan028 commented Mar 12, 2025

Please try AutoModelForCausalLM.from_pretrained(model_path), checking if it can be loaded by transformers without accessing the huggingface hub

@devops724
Copy link
Author

hard code address to local path in .cache/huggingface/hub/... working for me
thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants