Model with llama.cpp works, but not with llama-cpp-python

[gjmulder]

Discussed in #334

^{Originally posted by icarus0508 June 7, 2023}
Hi, i just build my llama.cpp model . it works fine on llama.cpp, but when i move the model to llama-cpp-python by following the code like:

nllm = LlamaCpp(
model_path="./ggml-model-q4_0.bin",
callback_manager=callback_manager,
verbose=True,
n_ctx=2048
)

it is always encount an error like this:

llama.cpp: loading model from ./ggml-model-q4_0.bin
error loading model: unrecognized tensor type 14

llama_init_from_file: failed to load model
2023-06-07 15:15:52.963 Uncaught app exception
Traceback (most recent call last):
File "C:\Users\IamUser\AppData\Local\Programs\Python\Python310\lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 565, in _run_script
exec(code, module.dict)
File "D:\PythonPlayground\NN-Playgroud\chatbot_NPC_llama.py", line 39, in
nllm = LlamaCpp(
File "pydantic\main.py", line 341, in pydantic.main.BaseModel.init
pydantic.error_wrappers.ValidationError: 1 validation error for LlamaCpp
root
Could not load Llama model from path: ./ggml-model-q4_0.bin. Received error (type=value_error)

llama.cpp version : master-ffb06a3

llama-cpp-python: v0.157

it will work if i use older version of model and use v0.149 of llama-cpp-python

but i will like to use the lastest version. How can i solve this?

Thank you

[gjmulder]

@icarus0508, I converted your Discussion into an issue.

This seems to be the same issue:

ggml-org/llama.cpp#1147

There are three ggml versions. You will want to use ggml v3 to avoid compatibility issues between the model and the release of llama-cpp-python.

Are you sure you're using the latest version, as it your limited description suggests you're trying to load a ggml v3 model with an older version llama-cpp-python?

The following code will tell you the version of model you are using. The same info is reported by llama.cpp. We should probably release this as a script:

def check_magic_and_version(filename):
    with open(filename, 'rb') as f:
        # Read the first 6 bytes from the file
        data = f.read(6)

    # Unpack the binary data, interpreting the first 4 bytes as a little-endian unsigned int
    # and the next 2 bytes as a little-endian unsigned short
    magic, version = struct.unpack('<I H', data)

    print(f"magic: 0x{magic:08x}, version: 0x{version:04x}, file: {filename}")

    return magic, version

To ensure you running the latest version of llama-cpp-python please reinstall using this command:

pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir

[icarus0508]

@gjmulder

Thanks for reply

I upgraded version to :

llama-cpp-python ver:0.1.59

llama.cpp ver:master-ffb06a3

after upgrade llama-cpp-python to v0.159

My alpaca (ggml-alpaca-7b-q4.bin) works fine but still llama(ggml-model-q4_0.bin) model can't be loaded().

it will occur a exception :

Exception has occurred: ValidationError
1 validation error for LlamaCpp
root
Could not load Llama model from path: ./models/7B/ggml-model-q4_0.bin. Received error exception: integer divide by zero (type=value_error)
File "D:\PythonPlayground\NN-Playgroud\TryLoadLLMModels.py", line 27, in
llm = LlamaCpp(
pydantic.error_wrappers.ValidationError: 1 validation error for LlamaCpp
root
Could not load Llama model from path: ./models/7B/ggml-model-q4_0.bin. Received error exception: integer divide by zero (type=value_error)

It will be fine for me to just use alpaca model, but I just wonder how to fix to use llama model ?

All models are converted using (llama.cpp ver:master-ffb06a3)

Thank you.

[gjmulder]

The model is likely the wrong version. In any case this is not a llama-cpp-python issue.

[ayttop]

it lama cpp python supporte flux1-dev-Q8_0.gguf