3B q4_0 open llama model running on a 4gb Pixel

[BarfingLemurs]

I just wanted to share that via termux 4gb of ram is enough to run this model, taking 1.9 gb of ram.

My hope is that I can use an older phone as a chatbot, using the talk-llama example found in https://github.com/ggerganov/whisper.cpp. I could leave my the phone plugged 24/7 only using 1.5W of power. Currently termux wont detect any audio, but you can run the talk-llama using 3b model on a pc, it's fast.

I also tried other 3b models running with an interface on kobold.cpp, it crashes due to the extra ram needed to load the interface.

[Green-Sky]

it crashes due to the extra ram needed to load the interface.

glorious.

[BarfingLemurs]

no room for luxuries, next I'll be fitting a 13b model on a portable usb headless system for use on a 8gb ram machine..ahaha