Update max_new_tokens slider in web gui to support at least 16k

[marmelade500]

[Flanua]

I don't think there is AI models that support that yet. You can use RoPE scaling for that if your model supports it.

P.S: More info here:
ggml-org/llama.cpp#2054

[berkut1]

@Flanua I think the OP is talking about token generation (like long respond), not context.

If this is equal to llama.embedding_length, then some models support 5120 or even 16k.

[marmelade500]

For anyone who wants a workaround, you can create a copy of the settings yaml, rename it to settings.yaml, and put your own custom settings in it. It will use those settings when the program is started.

[github-actions]

This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.