[Feature] add serve model with no huggingface request

[devops724]

Motivation

on run serve
CUDA_VISIBLE_DEVICES=0,1 lmdeploy serve api_server $1 --model-name image-processor --backend turbomind --server-port 7002 --model-format $2 --tp 2 --log-level INFO --session-len $3

i randomly get this error
File "/home/user/miniconda3/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/OpenGVLab/InternVL2_5-26B-MPO-AWQ/revision/main (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/user/miniconda3/bin/lmdeploy", line 8, in
sys.exit(run())
^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/lmdeploy/cli/entrypoint.py", line 31, in run
args.model_path = get_model(args.model_path, download_dir=download_dir, revision=revision)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/lmdeploy/utils.py", line 210, in get_model
model_path = snapshot_download(pretrained_model_name_or_path, ignore_patterns=['*.pth'], **download_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/_snapshot_download.py", line 155, in snapshot_download
repo_info = api.repo_info(repo_id=repo_id, repo_type=repo_type, revision=revision, token=token)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 2756, in repo_info
return method(
^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 2540, in model_info
r = get_session().get(path, headers=headers, timeout=timeout, params=params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/huggingface_hub/utils/_http.py", line 93, in send
return super().send(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/lib/python3.12/site-packages/requests/adapters.py", line 698, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/OpenGVLab/InternVL2_5-26B-MPO-AWQ/revision/main (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))"), '(Request ID: 31669d3e-6a2a-41e4-a68c-6ea611c3307a)')

Related resources

there should be a parameter model for working offline server in local network
prevent sending request to huggingface if model already downloaded

Additional context

No response

[lvhan028]

The logs indicate LMDeploy is trying to download model from huggingface hub.
Please check the value of $1, i.e., the model path, can be accessed and loaded successfully by transformers.

[devops724]

yes, it is in available , but that is because of it already downloaded and exist in cache and internet is not connected, it should ignore and use offline version instead of raise error

…

On Wed, Mar 12, 2025 at 2:48 PM Lyu Han ***@***.***> wrote: The logs indicate LMDeploy is trying to download model from huggingface hub. Please check the value of $1, i.e., the model path, can be accessed and loaded successfully by transformers. — Reply to this email directly, view it on GitHub <#3245 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKMDFEYYWBSA7PWQNO45THD2UAGGLAVCNFSM6AAAAABY3EK3IGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJXGQ2TANRZHA> . You are receiving this because you authored the thread.Message ID: ***@***.***> [image: lvhan028]*lvhan028* left a comment (InternLM/lmdeploy#3245) <#3245 (comment)> The logs indicate LMDeploy is trying to download model from huggingface hub. Please check the value of $1, i.e., the model path, can be accessed and loaded successfully by transformers. — Reply to this email directly, view it on GitHub <#3245 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKMDFEYYWBSA7PWQNO45THD2UAGGLAVCNFSM6AAAAABY3EK3IGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJXGQ2TANRZHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[lvhan028]

It INDEED access the local model path if the model exists and could be loaded by transformers

[lvhan028]

Please try AutoModelForCausalLM.from_pretrained(model_path), checking if it can be loaded by transformers without accessing the huggingface hub

[devops724]

hard code address to local path in .cache/huggingface/hub/... working for me
thanks