Accessing chat format functions directly

[FedericoMontana]

Hi! I'm trying to have access to see the formatted chats based on the messages. I'm trying to use low level access to what seems to be the function that formats the chat, but the output does not seem to be correct:

from llama_cpp import Llama
from llama_cpp.llama_chat_format import hf_tokenizer_config_to_chat_formatter


llm = Llama(
    model_path=".../models/microsoft/Phi-3-mini-4k-instruct-gguf/Phi-3-mini-4k-instruct-q4.gguf",
)

chat_template_value = llm.metadata.get('tokenizer.chat_template')
bos_token_value = llm.metadata.get('tokenizer.ggml.add_bos_token')
eos_token_value = llm.metadata.get('tokenizer.ggml.add_eos_token')

combined_dict = {
    'chat_template': chat_template_value,
    'bos_token': bos_token_value,
    'eos_token': eos_token_value
}

messages = [{'role': 'system', 'content': 'You are a friendly assistant'},  {'role': 'user', 'content': 'how are you?'}]

chat = hf_tokenizer_config_to_chat_formatter(combined_dict)
chat(messages).prompt

The output is this:
'true<|user|>\nhow are you?<|end|>\n<|assistant|>\n<|end|>\n'

For some reason it is not formatting the bos_token. I might be using incorrectly these low level functions, but I was wondering if there is a better way to have access to format random messages, and why am I doing wrong with my try, that I don't get the bos

Thank you!

[CISC]

You are not getting the bos/eos tokens, but rather the metadata telling llama.cpp if bos/eos tokens should be added or not. You can get the token IDs with llm.token_bos() and llm.token_eos(), but you have to convert those to text, and that is only available through the internal method llm._model.token_get_text(id), so user beware.

Since you are using hf_tokenizer_config_to_chat_formatter you should pass it the parsed tokenizer_config.json instead though.

[FedericoMontana]

Thank you so much! That worked. But still it seems I don't get to see the system message, which I guess is appended later on.

By chance, do you know a way to get the full formatted message assuming that the gguf got the metadata for that? Ideally without having to run create_chat_completion.

Thanks again :)

[CISC]

The chat template doesn't seem to support system messages, probably because the model doesn't, you'll have to print it out to see.

[FedericoMontana]

if you had the time, seems that this function might be used, but not sure how to use it (I tried but it exceeds my basic skills)

llama-cpp-python/llama_cpp/llama_cpp.py

Line 2709 in 83d6b26

def llama_chat_apply_template(

[CISC]

No, it's not useful to you, llama.cpp itself does not really support chat templates.