[FIXED] I wrote a script to merge lora thanks to slaren its done

[FNsi]

merge.py
it seems work. may stuck in embedding step? No clue now.

but while using convert.py

File "convert.py", line 1168, in
main()
File "onvert.py", line 1148, in main
model_plus = load_some_model(args.model)
File "convert.py", line 1076, in load_some_model
model_plus = merge_multifile_models(models_plus)
File "convert.py", line 583, in merge_multifile_models
model = merge_sharded([mp.model for mp in models_plus])
File "convert.py", line 562, in merge_sharded
return {name: convert(name) for name in names}
File "convert.py", line 562, in
return {name: convert(name) for name in names}
File "convert.py", line 537, in convert
lazy_tensors: List[LazyTensor] = [model[name] for model in models]
File "convert.py", line 537, in
lazy_tensors: List[LazyTensor] = [model[name] for model in models]
KeyError: 'embed_tokens.weight'

can someone try that script?

[slaren]

Using LlamaForCausalLM instead of LlamaModel seems to fix it.

[FNsi]

Using LlamaForCausalLM instead of LlamaModel seems to fix it.

It's not working in my env?

As I wrote in the last lines... "llamaforcusallm" no attribute "merge_and_unload"

[FNsi]

merge.txt

Using LlamaForCausalLM instead of LlamaModel seems to fix it.

It's not working in my env?

As I wrote in the last lines... "llamaforcusallm" no attribute "merge_and_unload"

So wired, I try it again and it work....
feel free to play with it.

[FNsi]

Using LlamaForCausalLM instead of LlamaModel seems to fix it.

Do you mind I creat a PR with it?

[Naozumi520]

May I ask, how can I use the model after merging? I gotpytorch_model-00001-of-00002.bin, pytorch_model-00002-of-00002.bin, config.json and generation_config.json after merging. Then, I used convert-llama-hf-to-gguf.py to covert to gguf. One problem is it returns error did not have the file tokenizer.model and refuse to process, so I copied it from the original base model (meta-llama/Llama-2-7b-chat-hf) and then quantized it into 4bit. But after that it didn't act like I expected. My lora is trained with Cantonese dataset, but the merge model didn't return even a word in cantonese. Any idea? :(

[FNsi]

May I ask, how can I use the model after merging? I gotpytorch_model-00001-of-00002.bin, pytorch_model-00002-of-00002.bin, config.json and generation_config.json after merging. Then, I used convert-llama-hf-to-gguf.py to covert to gguf. One problem is it returns error did not have the file tokenizer.model and refuse to process, so I copied it from the original base model (meta-llama/Llama-2-7b-chat-hf) and then quantized it into 4bit. But after that it didn't act like I expected. My lora is trained with Cantonese dataset, but the merge model didn't return even a word in cantonese. Any idea? :(

Sorry I didn't have any idea about GGUF now....

Might be able to convert it to ggml and then convert to gguf?