Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

是否支持VLM 输出logprobs #3243

Open
hxyghostor opened this issue Mar 12, 2025 · 6 comments
Open

是否支持VLM 输出logprobs #3243

hxyghostor opened this issue Mar 12, 2025 · 6 comments
Assignees

Comments

@hxyghostor
Copy link

Can VLLM(Qwen2-vl-2b-4bit) support logprobs output? I always get None

Image

@lvhan028
Copy link
Collaborator

Can you share the reproducible code?

@hxyghostor
Copy link
Author

hxyghostor commented Mar 12, 2025

server

lmdeploy serve api_server /question_classification/qwen2-vl-2b-4bit --server-port $PORT0 --backend turbomind --model-format awq --enable-prefix-caching --quant-policy 8

client

import requests
import base64

api_url = f"http://localhost:10516/v1/chat/completions"



image_path = ""
with open(image_path, "rb") as f:
            encoded_image = base64.b64encode(f.read())
encoded_image_text = encoded_image.decode("utf-8")
base64_qwen = f"data:image;base64,{encoded_image_text}"
data = {
            "model" : "/question_classification/qwen2-vl-2b-4bit",
            "messages":[
                {"role": "system", "content": "You are a helpful assistant."},
                {
                    "role": "user",
                    "content": [
                        {
                            "type": "image_url",
                            "image_url": {
                                "url": base64_qwen
                            },
                        },
                        {"type": "text", 
                         "text": ""},
                    ],
                },
            ],
            "logprobs": True,
            "top_logprobs": 3
        }

chat_response = requests.post(api_url, json=data).json()
print("Chat response:", chat_response)

@lvhan028
Copy link
Collaborator

Sorry, the pytorch engine hasn't supported output logprob yet.

@hxyghostor
Copy link
Author

Is there a time estimate for when tuibomind will support Qwen2 VL?

@lvhan028
Copy link
Collaborator

Probably at the end of this month. I am reviewing the #3164. Once it is merged to main branch, @irexyc can submit the implementation of qwen2-vl and qwen2.5-vl

@hxyghostor
Copy link
Author

OK,thanks for your reply.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants