You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.
3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
Checklist
Describe the bug
在测试超高rps的场景下lmdeploy性能时出现bug。benchmark选用lmdeploy/benchmark/profile_restful_api.py。
原因在于ShareGPT数据集中有prompt_len==0的情况,这可能是由于sharegpt采样时本身的数据就是空串。建议在采样时过滤掉空串的prompt。
修改:在代码中添加一个临时简单的绕过继续测试。(添加后不产生Warning)
Server报错
修改:在lmdeploy/lmdeploy/pytorch/paging/block_trie.py中添加判断(184、185行)。调用get_ref_count前检查leave_blocks是否为非空。
修改上述两项后,重新编译安装了lmdeploy。运行脚本。设置--random-input-len 8192或81920时能够正常运行。但是设置--random-input-len 9216时,出现ERROR,schedule()调度失败。
对此问题不知道如何解决。麻烦各位大佬帮忙看看问题可能出在哪儿,十分感谢Orz
Reproduction
server.sh
client.sh
Environment
Error traceback
The text was updated successfully, but these errors were encountered: