Llama 3.1 producing invalid output from Chat

[jmont-dev]

Llama 3.1 appears to have issues with the Chat endpoint, often producing empty or incoherent responses. The model appears to work with the standard generate endpoint.

Other models such as mistral and phi do appear to work without issues. There may be options or a special request structure that has to be made for Llama 3.

[jmont-dev]

This is resulting from providing a format value to the chat request, eg "format": "json". The API says that this should be acceptable but it appears to produce incoherent responses from some models. This tag has been commented out for now and the issue will be fed upstream to the Ollama team so they can address it or comment further.

The change removing the format field from chat requests was merged into main with #11.

[JG-Adams]

Removing format made both 3.1 and 3 work for me on MinGW Window. What is format supposed to do?

[jmont-dev]

Format represents the type of message that is returned from the Ollama server. Ollama currently only provides responses as JSON in HTTP replies to clients. The API lists this field as part of the specification and says that "json" is the only valid option. It's probably designed to future-proof the API in case they wanted to provide support for other types in the future (eg, YAML, TOML, or XML).

It definitely causes issues with some model responses, so they may be inadvertently passing the field on to the model itself which causes it to produce incoherent output. Marking this as closed for now and I'll push this up to the Ollama team to investigate in the base project.