Anyone keeping tabs on Vicuna, a new LLaMA-based model?

[jessejohnson]

Link to blog post, demo and GH: https://vicuna.lmsys.org/, https://chat.lmsys.org/, https://github.com/lm-sys/FastChat

This looks like the most capable LLaMA right now. They're yet to release the weights :)

[BetaDoggo]

Dep is pissed that they stole his name.
https://huggingface.co/chavinlo/vicuna

[GeorgeUCB]

It was bound to happen at some point.

[edmundronald]

So what's the news on this? Are the quantified weights available?

[jessejohnson]

There are ggml weights on 🤗uploaded just yesterday. Haven’t had the chance to try them yet

On Tue, 4 Apr 2023 at 02:23, edmundronald ***@***.***> wrote: So what's the news on this? Are the quantified weights available? — Reply to this email directly, view it on GitHub <#643 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJS2AIFDGFXB5YM42NW65LW7OAZHANCNFSM6AAAAAAWOAZEQQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Regards, Jesse Jojo Johnson http://www.jessejojojohnson.com/

[jxy]

Tried the one on huggingface hub, eachadea/ggml-vicuna-13b-4bit.

./main -m ./models/vicuna_13B/ggml-vicuna-13b-4bit.bin -t 4 -c 2048 -n 2048 --color -i --reverse-prompt '### Human:' -p '### Human: What is the relation between llama and vicuna?
### Assistant:'
main: seed = 1680621644
llama_model_load: loading model from './models/vicuna_13B/ggml-vicuna-13b-4bit.bin' - please wait ...
llama_model_load: n_vocab = 32001
llama_model_load: n_ctx   = 2048
llama_model_load: n_embd  = 5120
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 40
llama_model_load: n_layer = 40
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 13824
llama_model_load: n_parts = 2
llama_model_load: type    = 2
llama_model_load: ggml map size = 7759.84 MB
llama_model_load: ggml ctx size = 101.25 KB
llama_model_load: mem required  = 9807.93 MB (+ 1608.00 MB per state)
llama_model_load: loading tensors from './models/vicuna_13B/ggml-vicuna-13b-4bit.bin'
llama_model_load: model size =  7759.40 MB / num tensors = 363
llama_init_from_file: kv self size  = 1600.00 MB

system_info: n_threads = 4 / 8 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | VSX = 0 | 
main: interactive mode on.
Reverse prompt: '### Human:'
sampling: temp = 0.800000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.100000
generate: n_ctx = 2048, n_batch = 8, n_predict = 2048, n_keep = 0


== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to LLaMa.
 - If you want to submit another line, end your input in '\'.

 ### Human: What is the relation between llama and vicuna?
### Assistant: Llama and vicuña are two different species of animals that are closely related to each other. Both are members of the camelid family, which includes camels, llamas, alpacas, and vicuñas. However, the main difference between them is their size and physical characteristics.
Llama: The llama is a larger animal compared to the vicuña. It typically stands around 5 feet tall (1.5 meters) and weighs around 400-600 pounds (180-270 kilograms). They have a stocky build, with a thick fleece that can vary in color from black to white, depending on the breed. Llamas are known for their hardiness, intelligence, and ability to carry heavy loads over long distances.
Vicuña: The vicuña is a smaller animal than the llama. It typically stands around 3 feet tall (0.9 meters) and weighs around 150-200 pounds (68-91 kilograms). They have a slender build, with a fine, delicate fleece that is prized for its softness and warmth. Vicuñas are known for their extraordinary ability to produce fiber that is stronger than steel, yet incredibly lightweight and soft.
In terms of behavior, both animals are social and require the company of other members of their species. Llamas and vicuñas have a similar diet, which mainly consists of grasses, but they may also eat leaves and bark from trees and shrubs. They both have a long history of being domesticated for use as pack animals and for producing wool or fiber.
In summary, the llama and vicuña are related species within the camelid family, with the main difference being their size and physical characteristics. Llamas are larger and more hardy than vicuñas, while vicuñas produce finer and softer fiber that is highly prized for its properties.
### Human:

[edmundronald]

Disregard if dumb: It looks like there's a specific docker script called entrypoint.sh which is buggy here.
I'm not getting this error, but am compiling and running yesterday's llama.cpp from the repo direcly on my Macbook

[jmtatsch]

Hmm, I compiled from source on my M1 Macbook and it runs fine with

./main -m ./models/ggml-vicuna-13b-4bit.bin -n 256 --repeat_penalty 1.1 --color -i -r "### Human:" -f prompts/chat-with-vicuna.txt

and the same weights. So there seems to be some issue with the docker or the x64 platform.

[jmtatsch]

Docker issue is mentioned here #537

[thomasgatt]

@jmtatsch how's the performance on your Macbook? I'm running ggml-vicuna-13b-1.1-q4_0.bin on my M1 Pro 16gb ram and the performance is really slow. On my Windows Intel i7 8th Gen it's running like x5 faster...

[seunosewa]

Try to match the number of performance cores on your mac

[dbddv01]

First tries with vicunia 13b 4bit here. A zero-shot example here below. The answer is not that bad and written in the style of its big bro. Looks quite interesting.

[underlines]

you might want to try codealpaca fine-tuned gpt4all-alpaca-oa-codealpaca-lora-7b if you specifically ask coding related questions.

[JoseConseco]

Here is prompt for viscuna for llama.cpp
chat-with-vicuna.txt

I run it like so:
./main -m ./models/ggml-vicuna-13b-4bit.bin -n 256 --repeat_penalty 1.1 --color -i -r "### Human:" -f prompts/chat-with-vicuna.txt
I guess this is all that is needed for PR for Vicuna support in llama.cpp ...

Btw. any idea how do I redirect Vicuna output to some TTSpeech program ?

[x066it]

Check out whisper.cpp module named "talk-llama"

[ianscrivener]

@MaratZakirov,
there are 3 versions of GGML. 4 months ago might be an old version? There is no mention on GGML version.. so I guess that is a bad sign.

Suggest you get the latest llama.cpp code and make sure to get a GGMLv3 model

TheBloke does quality models - https://huggingface.co/TheBloke/vicuna-7B-v1.3-GGML/tree/main

[MaratZakirov]

I guess ggml version might be the issue. Thank you for link will try it

UPDATE: ggmlv3 was the issue thx again.

[underlines]

Yes, I keep tab on all llama descendants, under "models":

https://github.com/underlines/awesome-marketing-datascience/blob/master/awesome-ai.md

[dogjamboree]

FYI, It's missing the new Koala model from the same people who brought us Vicuna. Would love to see a version that could run under llama.cpp so I could check it out on my measly m2 pro processor...
https://bair.berkeley.edu/blog/2023/04/03/koala/

[underlines]

FYI, It's missing the new Koala model from the same people who brought us Vicuna. Would love to see a version that could run under llama.cpp so I could check it out on my measly m2 pro processor... https://bair.berkeley.edu/blog/2023/04/03/koala/

added the koala models and fine tuned versions and some third party models that are not based on llama.

[rg089]

Sorry if this is obvious, but is there a way currently to run the quantized Vicuna model in Python interactively on CPU (any bindings)? Or a stable way to call the executable from Python interactively?

[ghost]

llamacpp?

[Edinburgh020]

@rg089
download llama.cpp and run with https://huggingface.co/chharlesonfire/ggml-vicuna-7b-4bit .

./main -m ./ggml-vicuna-7b-q4_0.bin -n 256 --repeat_penalty 1.0 --color -i -r "User:" -f prompts/chat-with-bob.txt

hope this can help you.

[MaratZakirov]

llama.cpp: loading model from ../Vikuna/ggml-vicuna-7b-q4_0.bin
error loading model: unexpectedly reached end of file
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model '../Vikuna/ggml-vicuna-7b-q4_0.bin'

[dbmikus]

I'm getting the same error. @MaratZakirov were you able to figure out why it was failing to load the model? I downloaded it from https://huggingface.co/chharlesonfire/ggml-vicuna-7b-4bit/tree/main.

[ianscrivener]

This problem was diagnosed to an OLD GGML version (https://huggingface.co/chharlesonfire/ggml-vicuna-7b-4bit/tree/main.)

GGML v1 & v2 will not work with the latest code

Please download a current, GGMLv3, from [The Bloke](https://huggingface.co/TheBloke)

[Edinburgh020]

i found this vicuna-7b-4bit in hf. https://huggingface.co/chharlesonfire/ggml-vicuna-7b-4bit

and with chat-with-vicuna.txt (provided by @JoseConseco , from lm-sys/FastChat#364)

and here is my result:

[Edinburgh020]

And compare to chat-with-bob, we can find some difference.

chat-with-bob.txt

[oderwat]

I just use an additional -r "##" with Vicuna (vicuna-AlekseyKorshuk-7B-GPTQ-4bit-128g) and so far, I am happy with the results.

[thomasgatt]

Anyone getting very slow performance on llama.cpp on M1 Pro 16gb RAM? Running ggml-vicuna-13b-1.1-q4_0.bin from https://huggingface.co/eachadea/ggml-vicuna-13b-1.1/tree/main

[oderwat]

Anyone getting very slow performance on llama.cpp on M1 Pro 16gb RAM? Running ggml-vicuna-13b-1.1-q4_0.bin from https://huggingface.co/eachadea/ggml-vicuna-13b-1.1/tree/main

You should watch your swap activity. If it is swapping, it will be very slow. There may be just not enough free RAM left on your system for a 13b model.

[kharvd]

Seems to work with StableVicuna as well https://stability.ai/blog/stablevicuna-open-source-rlhf-chatbot

Here are the steps if someone finds it useful:

git clone https://huggingface.co/CarperAI/stable-vicuna-13b-delta
cd stable-vicuna-13b-delta
pip install torch tqdm transformers sentencepiece
python3 apply_delta.py --base-model-path <path-to-llama-weights-in-transformers-format> --target-model-path stable-vicuna-13b --delta-path .
cd llama.cpp
python convert-pth-to-ggml.py ./models/stable-vicuna-13b 1
./quantize ./models/stable-vicuna-13b/ggml-model-f16.bin ./my-models/stable-vicuna-13b/ggml-model-q4_0.bin q4_0

To convert LLaMA weights to the transformers format, I used this guide

git clone [email protected]:huggingface/transformers.git
cd transformers
pip install accelerate protobuf==3.20 sentencepiece tokenizers
python src/transformers/models/llama/convert_llama_weights_to_hf.py --input_dir <path_to_llama_weights> --model_size 13B --output_dir <path-to-llama-weights-in-transformers-format>

(this requires ~30 GB of RAM)

[KevinColemanInc]

If anyone else gets this: "main: failed to quantize model from './models/stable-vicuna-13b/ggml-model-f16.bin'"

You may need to rebuild your quantizer. In llama-cpp, run $ make

[warrenmwang]

Following your instructions there, I'm seeing ~30gb ram use just to load the 13B base model for me then I run OOM (I have 40 gb ram) when it starts loading the delta, anyone know a reason why or if im missing a step?

[rlancemartin]

How are folks running these models w/ reasonable latency? I've tested ggml-vicuna-7b-q4_0.bin and ggml-vicuna-13b-1.1-q4_0.bin on 16 GB RAM M1 Macbook Pro. Both are quite slow (as noted above for the 13b model). The above note suggests ~30GB RAM required for the 13b model. Obviously, the ability to run any of these models at all on a Macbook is very impressive, so I'm not really complaining, but am curious in practice what environment folks use for routine inference.

[perak]

Also, did you compiled it just by typing make or you are using some IDE which possibly compiled it with debug symbols? I got big performance difference between release and debug builds.

You can check directly your binary with dsymutil -s ./main | grep N_OSO which should give empty response on release build (and non-empty response on debug build).

[dogjamboree]

I'm not sure but I can tell you from personal experience in looking at timing (and listening to the fan!) that specifying the # of PERFORMANCE CORES vs total cores is the best strategy. So specifying 10 cores (total number of cores on the base m2 Pro ) vs 6 cores (number of performance cores) is significantly slower.

If I use --t 8, the fan the fan barely blows but with --t 6, I can barely hear myself talking :)

[perak]

@dogjamboree oh yes, I forgot to mention that: I got 10 cores on M1 Pro: 2 efficiency + 8 performance, that's why I am using -t 8. So @rlancemartin you should set number of threads to your number of performance cores for optimal results.

[rlancemartin]

You can check directly your binary with dsymutil -s ./main | grep N_OSO which should give empty response on release build (and non-empty response on debug build).

Thanks!

Confirmed I am using the release build (empty response w/ dsymutil -s ./main | grep N_OSO ).

I got 10 cores on M1 Pro: 2 efficiency + 8 performance, that's why I am using -t 8. So @rlancemartin you should set number of threads to your number of performance cores for optimal results.

I'm also on a M1 Pro w/ 16 GB RAM (6.5 GB used as baseline) and eight performance cores.

Running this across values of t and timing the query What is the largest city in Germany? ...

./main -m ./models/vicuna_7B/ggml-vicuna-7b-q4_0.bin -n 256 --repeat_penalty 1.0 --mlock --color -i -t xxx -r "User:" -f prompts/chat-with-bob.txt

... -t 6 looks best:
-t 10: > 30 sec (killed process)
-t 8: 18.7 sec
-t 6: 8.2 sec
-t 4: 9.7 sec
-t 2: 18.9 sec

[edwios]

Using ggml-vicuna-13b-q4_0.bin and t=6 (slightly better with t=8 on my machine [M1 Max 64GB RAM]) gave:

llama_print_timings:        load time =  5214.70 ms
llama_print_timings:      sample time =     0.32 ms /    13 runs   (    0.02 ms per run)
llama_print_timings: prompt eval time =  5526.28 ms /   108 tokens (   51.17 ms per token)
llama_print_timings:        eval time =   988.39 ms /    12 runs   (   82.37 ms per run)

(Edited: 7b->13b)

[dogjamboree]

Slightly off topic, but on the subject of mlock, is there any credence to the idea that pushing the limits of your ram and resorting to swap space on a modern SSD could burn it out relatively quickly is true? I was considering getting an external thunderbolt SSD just for swap because with 32gb ram these 30b parameter models really do seem to swap a lot, even though I have mlock on and they say they fit within my available ram (I guess the swapping is happening in other apps such as chrome, if I don't kill it).

[aidaho]

Slightly off topic, but on the subject of mlock, is there any credence to the idea that pushing the limits of your ram and resorting to swap space on a modern SSD could burn it out relatively quickly is true? I was considering getting an external thunderbolt SSD just for swap because with 32gb ram these 30b parameter models really do seem to swap a lot, even though I have mlock on and they say they fit within my available ram (I guess the swapping is happening in other apps such as chrome, if I don't kill it).

This isn't true. However, this is not a solution either. Swapping even on fast modern NVME will increase token generation time by orders of magnitude. BTW you should be able to fully fit 30b model in 32gb ram after some swapping out of the unnecessary programs. Regards, Serhii.

[FNsi]

If merge ton of loras, will the 13b beat the 65b version like llama beat gpt3?

[KevinColemanInc]

There isn't an inherit way to measure the performance of llama vs gpt3. Currently, most comparisons involve having the models take a test (like the bar exam or a medical exam) and compare the results.

I'd imagine you could take a single lora tuned to your test and it would beat gpt3. Maybe many loras can solve many tests (but there is some risk they could interfere with each other0.

[FNsi]

There isn't an inherit way to measure the performance of llama vs gpt3. Currently, most comparisons involve having the models take a test (like the bar exam or a medical exam) and compare the results.

I'd imagine you could take a single lora tuned to your test and it would beat gpt3. Maybe many loras can solve many tests (but there is some risk they could interfere with each other0.

And with cheapest price😁

[rlancemartin]

I've been using Vicuna for Question-Answering.

I'm using the py-bindings (llama-cpp-python) and LangChain.

My prompt template is:

template = """Use the following pieces of context to answer the question at the end. Use three sentences maximum. 
{context}
Question: {question}
Answer: Think step by step """

I'm initializing the model:

llm = LlamaCpp(
                model_path="/<path>/vicuna_13B/ggml-vicuna-13b-4bit.bin",
                callback_manager=callback_manager,
                verbose=True,
                n_threads=6,
                n_ctx=2048,
                use_mlock=True)

I notice it will answer questions in a rhetorical style with ### Human: and ### Assistant: in the answer:

### Human: xxx? 
### Assistant: xxx

Have others seen this and / or should I be using an alternative prompt?

[KevinColemanInc]

I noticed the same thing with my own prompt. I suspect it’s due to the fine tuning of the training data.

I don’t think they publicly released their exact dataset otherwise I’d spend the $1000 to modify it to how I want it.

[FNsi]

Just notice,

wizard-vicuna

[edmundronald]

It is going to be necessary to provide an automatic version conversion script. People who quantise a model aren't going to redo them and users don't have the ability to figure out compatibility issues

…

On Wed, Jul 26, 2023 at 9:58 AM MaratZakirov ***@***.***> wrote: I guess ggml version might be the issue. Thank you for link will try it — Reply to this email directly, view it on GitHub <#643 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABWGTOK5WYKHIJOH5ATCCDLXSDEY5ANCNFSM6AAAAAAWOAZEQQ> . You are receiving this because you commented.Message ID: <ggerganov/llama. ***@***.***>

[edmundronald]

The software should output a diagnostic explaining that it is dealing with a prior version. On Wed, Jul 26, 2023 at 1:37 PM edmund ronald ***@***.***> wrote:

…

It is going to be necessary to provide an automatic version conversion script. People who quantise a model aren't going to redo them and users don't have the ability to figure out compatibility issues On Wed, Jul 26, 2023 at 9:58 AM MaratZakirov ***@***.***> wrote: > I guess ggml version might be the issue. Thank you for link will try it > > — > Reply to this email directly, view it on GitHub > <#643 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABWGTOK5WYKHIJOH5ATCCDLXSDEY5ANCNFSM6AAAAAAWOAZEQQ> > . > You are receiving this because you commented.Message ID: <ggerganov/llama > .***@***.***> >