是否支持VLM 输出logprobs

[hxyghostor]

Can VLLM(Qwen2-vl-2b-4bit) support logprobs output? I always get None

[lvhan028]

Can you share the reproducible code?

[hxyghostor]

server

lmdeploy serve api_server /question_classification/qwen2-vl-2b-4bit --server-port $PORT0 --backend turbomind --model-format awq --enable-prefix-caching --quant-policy 8

client

import requests
import base64

api_url = f"http://localhost:10516/v1/chat/completions"



image_path = ""
with open(image_path, "rb") as f:
            encoded_image = base64.b64encode(f.read())
encoded_image_text = encoded_image.decode("utf-8")
base64_qwen = f"data:image;base64,{encoded_image_text}"
data = {
            "model" : "/question_classification/qwen2-vl-2b-4bit",
            "messages":[
                {"role": "system", "content": "You are a helpful assistant."},
                {
                    "role": "user",
                    "content": [
                        {
                            "type": "image_url",
                            "image_url": {
                                "url": base64_qwen
                            },
                        },
                        {"type": "text", 
                         "text": ""},
                    ],
                },
            ],
            "logprobs": True,
            "top_logprobs": 3
        }

chat_response = requests.post(api_url, json=data).json()
print("Chat response:", chat_response)

[lvhan028]

Sorry, the pytorch engine hasn't supported output logprob yet.

[hxyghostor]

Is there a time estimate for when tuibomind will support Qwen2 VL?

[lvhan028]

Probably at the end of this month. I am reviewing the #3164. Once it is merged to main branch, @irexyc can submit the implementation of qwen2-vl and qwen2.5-vl

[hxyghostor]

OK，thanks for your reply.