batch inference

[liuxiaohao-xn]

I have multiple prompts, I want to feed them all at once to the model to generate the output，can you tell me how to achieve it ?

[ilyalasy]

Same question here, it seems llama.cpp already supports batched inference.
See baby-llama.
See this discussion.

@abetlen It seems this usecase is very popular, what do you think?

[abetlen]

Currently the llama.h api does not support efficient batched inference. The babyllama example with batched inference uses the ggml api directly which this binding does not (I am working on a seperate project that does that but ggml repo is slightly outdated between ggml <-> llama.cpp). I'll take a look at some options for how to implement this though.

[ilyalasy]

Alright, thanks!

[origintopleft]

What kind of work would need to be done, either here or upstream in llama.cpp, to get batch inference fully working? Is there a roadmap anywhere already?

I've got some code here from work that needs my help working against Vicuna, and the only real obstacle left is getting batch processing working

[origintopleft]

I should probably clarify: I was asking because I was hoping to possibly contribute some help to the project 😅