-
Notifications
You must be signed in to change notification settings - Fork 508
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature] add serve model with no huggingface request #3245
Comments
The logs indicate LMDeploy is trying to download model from huggingface hub. |
yes, it is in available , but that is because of it already downloaded and
exist in cache and internet is not connected, it should ignore and use
offline version instead of raise error
…On Wed, Mar 12, 2025 at 2:48 PM Lyu Han ***@***.***> wrote:
The logs indicate LMDeploy is trying to download model from huggingface
hub.
Please check the value of $1, i.e., the model path, can be accessed and
loaded successfully by transformers.
—
Reply to this email directly, view it on GitHub
<#3245 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKMDFEYYWBSA7PWQNO45THD2UAGGLAVCNFSM6AAAAABY3EK3IGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJXGQ2TANRZHA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
[image: lvhan028]*lvhan028* left a comment (InternLM/lmdeploy#3245)
<#3245 (comment)>
The logs indicate LMDeploy is trying to download model from huggingface
hub.
Please check the value of $1, i.e., the model path, can be accessed and
loaded successfully by transformers.
—
Reply to this email directly, view it on GitHub
<#3245 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKMDFEYYWBSA7PWQNO45THD2UAGGLAVCNFSM6AAAAABY3EK3IGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJXGQ2TANRZHA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
It INDEED access the local model path if the model exists and could be loaded by transformers |
Please try |
hard code address to local path in .cache/huggingface/hub/... working for me |
Motivation
on run serve
CUDA_VISIBLE_DEVICES=0,1 lmdeploy serve api_server $1 --model-name image-processor --backend turbomind --server-port 7002 --model-format $2 --tp 2 --log-level INFO --session-len $3
i randomly get this error
File "/home/user/miniconda3/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/OpenGVLab/InternVL2_5-26B-MPO-AWQ/revision/main (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/user/miniconda3/bin/lmdeploy", line 8, in
sys.exit(run())
^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/lmdeploy/cli/entrypoint.py", line 31, in run
args.model_path = get_model(args.model_path, download_dir=download_dir, revision=revision)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/lmdeploy/utils.py", line 210, in get_model
model_path = snapshot_download(pretrained_model_name_or_path, ignore_patterns=['*.pth'], **download_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/_snapshot_download.py", line 155, in snapshot_download
repo_info = api.repo_info(repo_id=repo_id, repo_type=repo_type, revision=revision, token=token)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 2756, in repo_info
return method(
^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 2540, in model_info
r = get_session().get(path, headers=headers, timeout=timeout, params=params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/utils/_http.py", line 93, in send
return super().send(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/requests/adapters.py", line 698, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/OpenGVLab/InternVL2_5-26B-MPO-AWQ/revision/main (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))"), '(Request ID: 31669d3e-6a2a-41e4-a68c-6ea611c3307a)')
Related resources
there should be a parameter model for working offline server in local network
prevent sending request to huggingface if model already downloaded
Additional context
No response
The text was updated successfully, but these errors were encountered: