No such thing as CreateFile2? A MinGW problem

[JG-Adams]

On line 2858, inside httplib.h.
hFile_ = ::CreateFile2(wpath.c_str(), GENERIC_READ, FILE_SHARE_READ,
OPEN_EXISTING, NULL);

Is an error, '::CreateFile2' has not been declared; did you mean 'CreateFileW'? GCC

Also, in main.cpp sleep appeared to be mispelled for Sleep.

Is there a library I'm supposed to link to?
I would like to be able to try this. So please offer solution.

[jmont-dev]

Hey JG-Adams, Please see the Readme as it answers all of your questions. Specifically, the section at the top labeled Quick Start will show you how to use this library: https://github.com/jmont-dev/ollama-hpp?tab=readme-ov-file#quick-start

This is a header-only library. You just need to include the single-header include file and you can use it; no additional linking is required. This is also detailed in the Readme.

I've checked main.cpp in the main branch and latest release and they are already correct as "sleep". You may have changed this on your own during testing, but it appears correct in the library and has no issues building.

If you can give me some details on your platform and compiler I can help debug other issues that you're running into. This was tested and built on Rocky Linux 8.10 using gcc (GCC) 8.5.0. It should be broadly compatible with most other Linux distros; I haven't tested it on Windows but it may work there as well.

[JG-Adams]

I have read the readme. It's just saying compile the example. I tried both make and cmake they both have this error where CreateFile2 is undefined.
I'm on Window with MinGW GCC 13.2.
I don't have a clue what I'm supposed to be doing with this.

[JG-Adams]

Ok, found a solution for this issue. But another arise.
To fix the definition error. MinGW needed to have the macro set to use window10.
if (MINGW)
add_definitions(-D_WIN32_WINNT=0x0A00)
endif(MINGW)

It fix CreateFile2. But, not sleep. With it being Sleep it compile, but it has undefined error for some network.
So I must link to winsocket.
if (WIN32)
if (MINGW)
add_definitions(-D_WIN32_WINNT=0x0A00)
endif(MINGW)
link_libraries(wsock32 ws2_32)
endif (WIN32)

This compile. So, it is not truly a header only library. At least, not on window. But, that's easy to fix.

However. Running the program. I have no idea what is going on and what's supposed to happen, but, it fails on everything.
Here is the output I got:

[doctest] doctest version is "2.4.11"
[doctest] run with "--help" for options

F:/ollama-hpp-master/test/test.cpp:53:
TEST SUITE: Ollama Tests
TEST CASE: Load Model

F:/ollama-hpp-master/test/test.cpp:55: ERROR: CHECK( ollama::load_model("llama3:8b") ) THREW exception: "[json.exception.type_error.302] type must be boolean, but is null"

===============================================================================
F:/ollama-hpp-master/test/test.cpp:102:
TEST SUITE: Ollama Tests
TEST CASE: Basic Generation

F:/ollama-hpp-master/test/test.cpp:109: ERROR: CHECK( response.as_simple_string() == expected_response ) is NOT correct!
values: CHECK( The sky appears blue because of a phenomenon called Rayleigh scattering, named after the British physicist == What a great question!

The sky appears blue because of a phenomenon called Rayleigh scattering, )

===============================================================================
F:/ollama-hpp-master/test/test.cpp:124:
TEST SUITE: Ollama Tests
TEST CASE: Streaming Generation

F:/ollama-hpp-master/test/test.cpp:131: ERROR: CHECK( streamed_response == expected_response ) is NOT correct!
values: CHECK( The sky appears blue because of a phenomenon called Rayleigh scattering, named after the British physicist == What a great question!

The sky appears blue because of a phenomenon called Rayleigh scattering, )

===============================================================================
F:/ollama-hpp-master/test/test.cpp:134:
TEST SUITE: Ollama Tests
TEST CASE: Non-Singleton Generation

F:/ollama-hpp-master/test/test.cpp:144: ERROR: CHECK( response.as_simple_string() == expected_response ) is NOT correct!
values: CHECK( The sky appears blue because of a phenomenon called Rayleigh scattering, named after the British physicist == What a great question!

The sky appears blue because of a phenomenon called Rayleigh scattering, )

===============================================================================
F:/ollama-hpp-master/test/test.cpp:190:
TEST SUITE: Ollama Tests
TEST CASE: Generation with Image

F:/ollama-hpp-master/test/test.cpp:190: ERROR: test case THREW exception: Unable to open image file from path.

===============================================================================
F:/ollama-hpp-master/test/test.cpp:206:
TEST SUITE: Ollama Tests
TEST CASE: Generation with Multiple Images

F:/ollama-hpp-master/test/test.cpp:206: ERROR: test case THREW exception: Unable to open image file from path.

===============================================================================
/ollama-hpp-master/test/test.cpp:223:
TEST SUITE: Ollama Tests
TEST CASE: Chat with Image

F:/ollama-hpp-master/test/test.cpp:223: ERROR: test case THREW exception: Unable to open image file from path.

===============================================================================
[doctest] test cases: 20 | 13 passed | 7 failed | 0 skipped
[doctest] assertions: 19 | 15 passed | 4 failed |
[doctest] Status: FAILURE!

So, I have some question, What exactly is sleep? Does it function the same way as Sleep? What about thread_sleep?

[JG-Adams]

Whatever that test was supposed to do, something isn't consistent so it fail to pass. But, I'm able to have a working program with this.
However, in example main.cpp I'm not sure what blob is but it failed as well.

[jmont-dev]

Thanks for your feedback. A lot of the issues you ran into originate from your platform being different from the unix-based platform that I developed on. This was developed on Rocky Linux 8.9 and haven't had a chance to try it on Windows. I'll test it with minGW and Visual C++ when I have the opportunity.

sleep will be valid on unix-based platforms but it looks like the equivalent in Windows is Sleep. This can be replaced with std::this_thread::sleep_for in main.cpp which should be platform-agnostic and work with both.

This compiles on Linux with GCC without any special linkages. I don't see httplib documentation from YujiRose mentioning that winSock32 has to be linked on Windows; his documentation for httplib which is used in this library is here: https://github.com/yhirose/cpp-httplib. If wSock32 was required for linkage on your platform I can note that for other users.

Several of those tests are looking for precise text to be returned from a specific model with a specified seed; this was correct on creation but the base model weights have since been updated on huggingface which changed this output. I'm going to refactor these tests so that they don't rely on matching an exact response to be valid.

Blobs are files used to store model weights and parameters that are used in Ollama. I don't think any user will need to manually interact with these but they were included in the Ollama API spec so I added support for them. Even when I structure the query as stated in the API I'm getting a failure response from the server; this may be an issue upstream in Ollama or there may be something else going on. This will be addressed with lower priority since it isn't a feature that will be frequently used.

I have an upcoming release targeting the new Ollama tool support in 0.3.0 and will factor in some of your recommendations for changes. Thanks for doing some of the pathfinding to get this tested on Windows.

[jmont-dev]

Glad to hear you were able to get it working though. Let me know if there are any other issues that you find during testing.

[JG-Adams]

Thank you for making this! It's easier than Boost::asio :)

Is the function, chat supposed to return an AI response? Because it's not outputting anything.
std::cout << ollama::chat(Model, messages) << std::endl;

It's just outputting a bunch of newline.
Chat1 {"format":"json","keep_alive":"5m","messages":[{"content":"What are nimbus clouds?","role":"user"}],"model":"llama3.1:latest","stream":false}
{"model":"llama3.1:latest","created_at":"2024-08-10T06:57:10.747976683Z","message":{"role":"assistant","content":"{"small":"true"} \n\n \t\n\n\n\n \t\n\n\n\n \t\n\n\n\n \t\n\n\n\n
\t\n\n\n\n \t\n\n\n\n \t\n\n\n\n \t\n\n\n\n \t\n\n\n\n \t\n\n\n\n"},"done":false}
{"small":"true"}

Oddly enough, after running it a couple time there is a small chance it would actually output something.

Chat1 {"format":"json","keep_alive":"5m","messages":[{"content":"What are nimbus clouds?","role":"user"}],"model":"llama3.1:latest","stream":false}
{"model":"llama3.1:latest","created_at":"2024-08-10T06:56:15.77064905Z","message":{"role":"assistant","content":"{ \n "nimbus cloud": "Nimbus clouds are a type of dark, rain-bearing
cloud that belongs to the cumulus family. They can grow tall enough to develop into towering vertical growth with flat bases and rounded tops." \n }"},"done_reason":"stop","done":true,"total_duration":946954355,"load_duration":13119863,"prompt_eval_count":15,"prompt_eval_duration":31860000,"eval_count":51,"eval_duration":853889000}
{
"nimbus cloud": "Nimbus clouds are a type of dark, rain-bearing cloud that belongs to the cumulus family. They can grow tall enough to develop into towering vertical growth with flat bases and rounded tops."
}

I don't get why this is happening.

[JG-Adams]

I even got this.

{"smallGrid":"true","smallScreen":{"width":480,"height":200},"bigGrid":"false","bigScreen":{"width":1600,"height":1200},"articleType":"article" , "content":"

Nimbus clouds, also known as dark rain clouds or cumulonimbus clouds, are tall, dense, and dark clouds that can produce heavy precipitation, thunderstorms, and even tornadoes. They form when warm, moist air rises rapidly into cold air, creating towering vertical growth.

These clouds have a characteristic anvil-shaped base and a flat, anvil-like top. They can reach heights of over 10,000 meters (33,000 feet), making them the tallest type of cloud. Nimbus clouds are associated with severe weather conditions, including heavy rain, hail, lightning, and strong winds.

They
are formed in association with fronts or low-pressure systems and are commonly found near mountains or coastlines where warm air is forced to rise. The anvil-shaped base of the cloud indicates that it has reached its maximum vertical growth and is beginning to spread out horizontally, which is a sign of instability in the atmosphere.

Nimbus clouds can be further classified into different types based on their height, shape, and other characteristics. They are an important part of the weather cycle and play a crucial role in shaping our climate and environment.

","@type":"Article" , "image":{"@type":"ImageObject","url":"https://cdn.vox-cdn.com/thumbor/4fQgCtqTmzXK1xwZsUv8V7WJ6I=/0x0:1203x675/1600×900/filters:format(jpeg):quality(90)/crop/0x0:1203x675/https://cdn.vox-cdn.com/thumbor/4fQgCtqTmzXK1xwZsUv8V7WJ6I=/0x0:1203x675/1600×900/filters:format(jpeg):quality(90)/crop/0x0:1203x675/https://cdn.vox-cdn.com/uploads/2022/06/164f4e2d-8d41-4067-b5c6-a9dd44b0f1eb-1140.jpg","width":1203,"height":675}
}

Why is it trying to give me an image and link? This is llama3.1 I have no clue if it was supposed to have that capability or how to handle it.

[jmont-dev]

Yes, the chat endpoint should return a response just like generate. It looks like the issue is specific to the model; llama3.1/3 is returning no response or malformed responses from the server while other models look good. I'm not sure why this is happening; the query to the server looks good but ollama is immediately returning a blank reply containing the stop flag. I'll do some investigating and will talk to the folks in the main Ollama project to see if anyone else is seeing the same issues. It may be that a special flag or option has to be set for llama3.

In the meantime, the generate endpoint appears to work fine with 3.1. You can still structure your prompt as a question/answer series and generate an assistant reply using that method.

This is the chat response that I get from mistral, which looks good (debug logging for the server request and reply included):

ollama::message message1("user", "What are nimbus clouds?");
std::cout << ollama::chat("mistral", message1) << std::endl;

{"format":"json","keep_alive":"5m","messages":[{"content":"What are nimbus clouds?","role":"user"}],"model":"mistral","stream":false}
{"model":"mistral","created_at":"2024-08-10T15:08:22.136943919Z","message":{"role":"assistant","content":"{\n"question": "What are nimbus clouds?",\n"answer": "Nimbus clouds are clouds that are characterized by their flat, featureless appearance and the presence of rain or snow. They are also known as stratus or stratocumulus clouds. Nimbus clouds form at low to mid-level altitudes and can cover the entire sky, creating overcast conditions. They are typically associated with light precipitation, such as drizzle or light snow."\n}"},"done_reason":"stop","done":true,"total_duration":3522945962,"load_duration":2603335962,"prompt_eval_count":17,"prompt_eval_duration":16184000,"eval_count":103,"eval_duration":902607000}
{
"question": "What are nimbus clouds?",
"answer": "Nimbus clouds are clouds that are characterized by their flat, featureless appearance and the presence of rain or snow. They are also known as stratus or stratocumulus clouds. Nimbus clouds form at low to mid-level altitudes and can cover the entire sky, creating overcast conditions. They are typically associated with light precipitation, such as drizzle or light snow."
}

Multi-message chat also looks good with mistral:

ollama::message message1("user", "What are nimbus clouds?");
ollama::message message2("assistant", "Nimbus clouds are dense, moisture-filled clouds that produce rain.");
ollama::message message3("user", "What are some other kinds of clouds?");
ollama::messages messages = {message1, message2, message3};
std::cout << ollama::chat("mistral", messages) << std::endl;

{"format":"json","keep_alive":"5m","messages":[{"content":"What are nimbus clouds?","role":"user"},{"content":"Nimbus clouds are dense, moisture-filled clouds that produce rain.","role":"assistant"},{"content":"What are some other kinds of clouds?","role":"user"}],"model":"mistral","stream":false}
{"model":"mistral","created_at":"2024-08-10T15:08:25.159632956Z","message":{"role":"assistant","content":"{\n "Cirrus": "High, thin, and wispy clouds made of ice crystals. They often indicate fair weather but can also signal an approaching storm.",\n "Cumulus": "Large, puffy clouds with flat bases that resemble cauliflower. They are associated with fair weather but can also turn into thunderstorm clouds if conditions become unstable.",\n "Stratus": "Low-level, gray, and featureless clouds that often cover the sky completely. They are typically associated with overcast and dreary weather.",\n "Nimbostratus": "Dark, thick, and layered clouds that produce continuous rain or snowfall. They are often associated with bad weather and stormy conditions.",\n "Cumulonimbus": "Towering vertical clouds with a flat top and a thunderhead at the base. They are associated with severe storms, including tornadoes.",\n "Altocumulus": "Mid-level clouds that appear as rounded masses or rolls of gray or white. They can indicate an approaching storm or signal stable weather conditions.",\n "Altostratus": "Gray and featureless clouds that cover most of the sky at mid-level. They are typically associated with overcast and drizzly weather.",\n "Stratocumulus": "Low-level, gray, and featureless clouds that often appear in rows or sheets. They can indicate stable weather conditions but can also turn into thunderstorm clouds if conditions become unstable."\n}"},"done_reason":"stop","done":true,"total_duration":3021734510,"load_duration":5186806,"prompt_eval_count":48,"prompt_eval_duration":17938000,"eval_count":322,"eval_duration":2874778000}
{
"Cirrus": "High, thin, and wispy clouds made of ice crystals. They often indicate fair weather but can also signal an approaching storm.",
"Cumulus": "Large, puffy clouds with flat bases that resemble cauliflower. They are associated with fair weather but can also turn into thunderstorm clouds if conditions become unstable.",
"Stratus": "Low-level, gray, and featureless clouds that often cover the sky completely. They are typically associated with overcast and dreary weather.",
"Nimbostratus": "Dark, thick, and layered clouds that produce continuous rain or snowfall. They are often associated with bad weather and stormy conditions.",
"Cumulonimbus": "Towering vertical clouds with a flat top and a thunderhead at the base. They are associated with severe storms, including tornadoes.",
"Altocumulus": "Mid-level clouds that appear as rounded masses or rolls of gray or white. They can indicate an approaching storm or signal stable weather conditions.",
"Altostratus": "Gray and featureless clouds that cover most of the sky at mid-level. They are typically associated with overcast and drizzly weather.",
"Stratocumulus": "Low-level, gray, and featureless clouds that often appear in rows or sheets. They can indicate stable weather conditions but can also turn into thunderstorm clouds if conditions become unstable."
}

I'll track this in another issue.

[jmont-dev]

I don't think Llama 3.1 is supposed to respond with links or images. That looks like hallucinated text with the model just regurgitating something similar to the web pages it trained on, probably a symptom of improper formatting between ollama and the model when using the chat endpoint. It could also be that the default options for the model have something like high temperature or top-k which will result in more random and less structured output. I'll do some experimenting and see if I can reign it in for Llama 3.1 so that the chat endpoint is producing reliable output like generate does.

[jmont-dev]

The issues appears to be caused by the format tag being included with the chat requests. The Ollama API says that "format' :'json' is the only acceptable value for this currently. It looks like including this tag with some models such as Llama 3 and Gemma 2 causes Ollama to produce incoherent text. This is likely a bug in Ollama and I'll refer upstream to jmorganca and the other Ollama devs to take a look when they can.

I've addressed the issue by removing the formatting tag entirely from the requests made to the chat endpoint. This was merged with #11 and is included in the maintenance release 0.9.1 that was just issued. If you comment out line 253 in Ollama.hpp you should get normal behavior from chat when using Llama 3:

//(*this)["format"] = format; // Commented out as providing the format causes issues with some models.

Tracking this in #9 and will close once I hear back from the Ollama devs.

[JG-Adams]

Very interesting. Thank you for looking into this! You're doing great!
I think it's safe to say this is not a MinGW problem anymore, since it is now solved. I will close this.