[User] Unable to Finetune Llama 2 70B #3644

mrroll · 2023-10-16T14:29:16Z

Prerequisites

Please answer the following questions for yourself before submitting an issue.

I am running the latest code. Development is very rapid so there are no tagged versions as of now.
I carefully followed the README.md.
I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

fineunteing Llama 2 70B should succeed

Current Behavior

fineunteing Llama 2 70B fails with

ggml_allocr_alloc: not enough space in the buffer (needed 1048576000, largest block available 939524096)
GGML_ASSERT: ggml-alloc.c:148: !"not enough space in the buffer"
Aborted (core dumped)

I should add that finetuning Llama 2 13B works.

Environment and Context

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

System Memory

(base) user@server:/srv/shared$ free -h
               total        used        free      shared  buff/cache   available
Mem:           377Gi       2.1Gi       8.4Gi       1.0Mi       367Gi       373Gi
Swap:             0B          0B          0B

Physical (or virtual) hardware you are using, e.g. for Linux:

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  96
  On-line CPU(s) list:   0-95
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Xeon(R) CPU @ 2.20GHz
    CPU family:          6
    Model:               85
    Thread(s) per core:  2
    Core(s) per socket:  24
    Socket(s):           2
    Stepping:            7
    BogoMIPS:            4400.36
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflu
                         sh mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good
                          nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16
                          pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor 
                         lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced f
                         sgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx51
                         2dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsave
                         c xgetbv1 xsaves arat avx512_vnni md_clear arch_capabilities
Virtualization features: 
  Hypervisor vendor:     KVM
  Virtualization type:   full
Caches (sum of all):     
  L1d:                   1.5 MiB (48 instances)
  L1i:                   1.5 MiB (48 instances)
  L2:                    48 MiB (48 instances)
  L3:                    77 MiB (2 instances)
NUMA:                    
  NUMA node(s):          2
  NUMA node0 CPU(s):     0-23,48-71
  NUMA node1 CPU(s):     24-47,72-95
Vulnerabilities:         
  Gather data sampling:  Unknown: Dependent on hypervisor status
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Mitigation; Clear CPU buffers; SMT Host state unknown
  Meltdown:              Not affected
  Mmio stale data:       Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
  Retbleed:              Mitigation; Enhanced IBRS
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequ
                         ence
  Srbds:                 Not affected
  Tsx async abort:       Mitigation; Clear CPU buffers; SMT Host state unknown

Operating System, e.g. for Linux:

Linux 6.2.0-1016-gcp #18~22.04.1-Ubuntu SMP Fri Sep 29 04:56:44 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

SDK version, e.g. for Linux:

Python 3.10.9
GNU Make 4.3
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Failure Information (for bugs)

ggml_allocr_alloc: not enough space in the buffer (needed 1048576000, largest block available 939524096)
GGML_ASSERT: ggml-alloc.c:148: !"not enough space in the buffer"
Aborted (core dumped)

Steps to Reproduce

The following steps assume that:

You have miniconda installed and the base environment is loaded.
You have access to the Llama 2 base model from Meta.
In addition, you have already downloaded the Llama 2 70B model and placed it in ./models/llama-2-70b.
You have downloaded shakespeare.txt and placed it in the root of the git repository.

Clone llama.cpp.
cd into the directory where llama.cpp was cloned.
Run make
Create a miniconda environment called llama.

conda create -yn llama python=3.10.9

Switch to the miniconda environment you just created.

conda activate llama

Install dependencies

pip install --upgrade --requirement requirements.txt
pip install --upgrade torch transformers

Convert the Llama 2 70B model into the GGUF format.

python convert.py ./models/llama-2-70b

Quantize the converted model

./quantize ./models/llama-2-70b/ggml-model-f16.gguf ./models/llama-2-70b/ggml-model-q8_0.gguf q8_0

Attempt to finetune the model. For testing, I used this text file

./finetune \
    --model-base ./models/llama-2-70b/ggml-model-q8_0.gguf \
    --checkpoint-in llama-2-70b-shakespeare-LATEST.gguf \
    --checkpoint-out llama-2-70b-shakespeare-ITERATION.gguf \
    --lora-out llama-2-70b-shakespeare-ITERATION.bin \
    --train-data "./shakespeare.txt" \
    --save-every 10 \
    --threads 48 --adam-iter 150 --batch 4 --ctx 64 \
    --use-checkpointing

Failure Logs

The logs are too long to include as a comment. Instead, I am attaching them here. You'll also find that I ran a finetune on Llama 13B just to demonstrate that it's working.

error.log

The text was updated successfully, but these errors were encountered:

RedAndr · 2023-10-16T15:26:16Z

How much memory do you have?

mrroll · 2023-10-16T23:32:14Z

How much memory do you have?

@RedAndr Sorry I completely forgot to add this in. I'm editing this into the original issue as well

(base) user@server:/srv/shared$ free -h
               total        used        free      shared  buff/cache   available
Mem:           377Gi       2.1Gi       8.4Gi       1.0Mi       367Gi       373Gi
Swap:             0B          0B          0B

QueryType · 2023-10-25T15:36:44Z

I am getting the similar error, on mac mini M2, 24GB or memory. Base model: openllama-3b-v2

main: lora_size = 54798560 bytes (52.3 MB)
main: opt_size  = 81693904 bytes (77.9 MB)
main: opt iter 0
main: input_size = 32769056 bytes (31.3 MB)
main: compute_size = 2785785440 bytes (2656.7 MB)
main: evaluation order = RIGHT_TO_LEFT
ggml_allocr_alloc: not enough space in the buffer (needed 409600000, largest block available 339804192)
GGML_ASSERT: ggml-alloc.c:148: !"not enough space in the buffer"
./train.sh: line 1:  1108 Abort trap: 6           /Volumes/d/apps/llama.cpp/llama.cpp/finetune --model-base /Volumes/d/apps/aimodels/others/openllama-3b-v2/openllama-3b-v2.q8_0.gguf --train-data shakespeare.txt --lora-out lora.gguf --save-every 0 --threads 8 --ctx 256 --rope-freq-base 10000 --rope-freq-scale 1.0 --batch 1 --grad-acc 1 --adam-iter 256 --adam-alpha 0.001 --lora-r 4 --lora-alpha 4 --use-checkpointing --use-flash --sample-start "\n" --escape --include-sample-start --seed 1

KerfuffleV2 · 2023-10-25T19:53:24Z

This doesn't directly help you, but the error isn't related to how much memory you have available. I believe it means that the maximum context size didn't get calculated correctly. So not a user error.

QueryType · 2023-10-26T00:51:57Z

This doesn't directly help you, but the error isn't related to how much memory you have available. I believe it means that the maximum context size didn't get calculated correctly. So not a user error.

Thanks for the response @KerfuffleV2 So I believe there's some bug to be fixed, but curious if it impacts everyone trying to use finetune or just some combination of input?

mrroll · 2023-10-26T00:57:22Z

Hey folks! On 6961c4b, I'm getting a new error:

GGML_ASSERT: ggml-alloc.c:240: alloc->n_free_blocks < MAX_FREE_BLOCKS && "out of free blocks"

QueryType · 2023-10-26T17:02:14Z

I am getting the similar error, on mac mini M2, 24GB or memory. Base model: openllama-3b-v2

main: lora_size = 54798560 bytes (52.3 MB)
main: opt_size  = 81693904 bytes (77.9 MB)
main: opt iter 0
main: input_size = 32769056 bytes (31.3 MB)
main: compute_size = 2785785440 bytes (2656.7 MB)
main: evaluation order = RIGHT_TO_LEFT
ggml_allocr_alloc: not enough space in the buffer (needed 409600000, largest block available 339804192)
GGML_ASSERT: ggml-alloc.c:148: !"not enough space in the buffer"
./train.sh: line 1:  1108 Abort trap: 6           /Volumes/d/apps/llama.cpp/llama.cpp/finetune --model-base /Volumes/d/apps/aimodels/others/openllama-3b-v2/openllama-3b-v2.q8_0.gguf --train-data shakespeare.txt --lora-out lora.gguf --save-every 0 --threads 8 --ctx 256 --rope-freq-base 10000 --rope-freq-scale 1.0 --batch 1 --grad-acc 1 --adam-iter 256 --adam-alpha 0.001 --lora-r 4 --lora-alpha 4 --use-checkpointing --use-flash --sample-start "\n" --escape --include-sample-start --seed 1

This issue comes when I have --batch 1, any other value works fine.

mrroll · 2023-10-27T03:33:33Z

I am getting the similar error, on mac mini M2, 24GB or memory. Base model: openllama-3b-v2

main: lora_size = 54798560 bytes (52.3 MB)
main: opt_size  = 81693904 bytes (77.9 MB)
main: opt iter 0
main: input_size = 32769056 bytes (31.3 MB)
main: compute_size = 2785785440 bytes (2656.7 MB)
main: evaluation order = RIGHT_TO_LEFT
ggml_allocr_alloc: not enough space in the buffer (needed 409600000, largest block available 339804192)
GGML_ASSERT: ggml-alloc.c:148: !"not enough space in the buffer"
./train.sh: line 1:  1108 Abort trap: 6           /Volumes/d/apps/llama.cpp/llama.cpp/finetune --model-base /Volumes/d/apps/aimodels/others/openllama-3b-v2/openllama-3b-v2.q8_0.gguf --train-data shakespeare.txt --lora-out lora.gguf --save-every 0 --threads 8 --ctx 256 --rope-freq-base 10000 --rope-freq-scale 1.0 --batch 1 --grad-acc 1 --adam-iter 256 --adam-alpha 0.001 --lora-r 4 --lora-alpha 4 --use-checkpointing --use-flash --sample-start "\n" --escape --include-sample-start --seed 1

This issue comes when I have --batch 1, any other value works fine.

@QueryType So you got it working? If so, what specific arguments did you use?

QueryType · 2023-10-27T13:06:01Z

I am getting the similar error, on mac mini M2, 24GB or memory. Base model: openllama-3b-v2

main: lora_size = 54798560 bytes (52.3 MB)
main: opt_size  = 81693904 bytes (77.9 MB)
main: opt iter 0
main: input_size = 32769056 bytes (31.3 MB)
main: compute_size = 2785785440 bytes (2656.7 MB)
main: evaluation order = RIGHT_TO_LEFT
ggml_allocr_alloc: not enough space in the buffer (needed 409600000, largest block available 339804192)
GGML_ASSERT: ggml-alloc.c:148: !"not enough space in the buffer"
./train.sh: line 1:  1108 Abort trap: 6           /Volumes/d/apps/llama.cpp/llama.cpp/finetune --model-base /Volumes/d/apps/aimodels/others/openllama-3b-v2/openllama-3b-v2.q8_0.gguf --train-data shakespeare.txt --lora-out lora.gguf --save-every 0 --threads 8 --ctx 256 --rope-freq-base 10000 --rope-freq-scale 1.0 --batch 1 --grad-acc 1 --adam-iter 256 --adam-alpha 0.001 --lora-r 4 --lora-alpha 4 --use-checkpointing --use-flash --sample-start "\n" --escape --include-sample-start --seed 1

This issue comes when I have --batch 1, any other value works fine.

@QueryType So you got it working? If so, what specific arguments did you use?

finetune --model-base /Volumes/d/apps/aimodels/others/openllama-3b-v2/openllama-3b-v2.q8_0.gguf --train-data shakespeare.txt --lora-out lora.gguf --save-every 0 --threads 10 --ctx 256 --batch 32 --use-checkpointing --use-flash --sample-start "\n" --escape

Yes, but it is too slow and fires up the CPU to 100 deg. C! Also swap got activated.

mrroll · 2023-10-27T15:19:57Z

Nice! Thanks! I myself do not have an issue with lower parameter models like 13B, but 70B just doesn't want to start at all with the error:

GGML_ASSERT: ggml-alloc.c:240: alloc->n_free_blocks < MAX_FREE_BLOCKS && "out of free blocks"

slaren · 2023-10-27T15:22:13Z

You can probably workaround that problem by increasing MAX_FREE_BLOCKS in ggml-alloc.c. But we need a better long term solution, the value is already too big as it is.

mrroll · 2023-10-30T12:00:47Z

You can probably workaround that problem by increasing MAX_FREE_BLOCKS in ggml-alloc.c. But we need a better long term solution, the value is already too big as it is.

I actually tried that previously -- increasing it to 512. I now tried increasing it to 1024, 2048, and 4096. On every change, I tried to run a finetune and end up with an error message terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc

Log

/srv/shared/llama.cpp/finetune main: seed: 1698665668
main: model base = '/srv/shared/ll ggml_init_cublas: GGML_CUDA_FORCE_MMQ: ggml_init_cublas: CUDA_USE_TENSOR_CORES: ggml_init_cublas: found 4 Device 0: NVIDIA L4, compute Device 1: NVIDIA L4, compute Device 2: NVIDIA L4, compute Device 3: NVIDIA L4, compute llama_model_loader: loaded llama_model_loader: - tensor 0: llama_model_loader: - tensor 1: llama_model_loader: - tensor 2: llama_model_loader: - tensor 3: llama_model_loader: - tensor 4: llama_model_loader: - tensor 5: llama_model_loader: - tensor 6: llama_model_loader: - tensor 7: llama_model_loader: - tensor 8: llama_model_loader: - tensor 9: llama_model_loader: - tensor 10: llama_model_loader: - tensor 11: llama_model_loader: - tensor 12: llama_model_loader: - tensor 13: llama_model_loader: - tensor 14: llama_model_loader: - tensor 15: llama_model_loader: - tensor 16: llama_model_loader: - tensor 17: llama_model_loader: - tensor 18: llama_model_loader: - tensor 19: llama_model_loader: - tensor 20: llama_model_loader: - tensor 21: llama_model_loader: - tensor 22: llama_model_loader: - tensor 23: llama_model_loader: - tensor 24: llama_model_loader: - tensor 25: llama_model_loader: - tensor 26: llama_model_loader: - tensor 27: llama_model_loader: - tensor 28: llama_model_loader: - tensor 29: llama_model_loader: - tensor 30: llama_model_loader: - tensor 31: llama_model_loader: - tensor 32: llama_model_loader: - tensor 33: llama_model_loader: - tensor 34: llama_model_loader: - tensor 35: llama_model_loader: - tensor 36: llama_model_loader: - tensor 37: llama_model_loader: - tensor 38: llama_model_loader: - tensor 39: llama_model_loader: - tensor 40: llama_model_loader: - tensor 41: llama_model_loader: - tensor 42: llama_model_loader: - tensor 43: llama_model_loader: - tensor 44: llama_model_loader: - tensor 45: llama_model_loader: - tensor 46: llama_model_loader: - tensor 47: llama_model_loader: - tensor 48: llama_model_loader: - tensor 49: llama_model_loader: - tensor 50: llama_model_loader: - tensor 51: llama_model_loader: - tensor 52: llama_model_loader: - tensor 53: llama_model_loader: - tensor 54: llama_model_loader: - tensor 55: llama_model_loader: - tensor 56: llama_model_loader: - tensor 57: llama_model_loader: - tensor 58: llama_model_loader: - tensor 59: llama_model_loader: - tensor 60: llama_model_loader: - tensor 61: llama_model_loader: - tensor 62: llama_model_loader: - tensor 63: llama_model_loader: - tensor 64: llama_model_loader: - tensor 65: llama_model_loader: - tensor 66: llama_model_loader: - tensor 67: llama_model_loader: - tensor 68: llama_model_loader: - tensor 69: llama_model_loader: - tensor 70: llama_model_loader: - tensor 71: llama_model_loader: - tensor 72: llama_model_loader: - tensor 73: llama_model_loader: - tensor 74: llama_model_loader: - tensor 75: llama_model_loader: - tensor 76: llama_model_loader: - tensor 77: llama_model_loader: - tensor 78: llama_model_loader: - tensor 79: llama_model_loader: - tensor 80: llama_model_loader: - tensor 81: llama_model_loader: - tensor 82: llama_model_loader: - tensor 83: llama_model_loader: - tensor 84: llama_model_loader: - tensor 85: llama_model_loader: - tensor 86: llama_model_loader: - tensor 87: llama_model_loader: - tensor 88: llama_model_loader: - tensor 89: llama_model_loader: - tensor 90: llama_model_loader: - tensor 91: llama_model_loader: - tensor 92: llama_model_loader: - tensor 93: llama_model_loader: - tensor 94: llama_model_loader: - tensor 95: llama_model_loader: - tensor 96: llama_model_loader: - tensor 97: llama_model_loader: - tensor 98: llama_model_loader: - tensor 99: llama_model_loader: - tensor 100: llama_model_loader: - tensor 101: llama_model_loader: - tensor 102: llama_model_loader: - tensor 103: llama_model_loader: - tensor 104: llama_model_loader: - tensor 105: llama_model_loader: - tensor 106: llama_model_loader: - tensor 107: llama_model_loader: - tensor 108: llama_model_loader: - tensor 109: llama_model_loader: - tensor 110: llama_model_loader: - tensor 111: llama_model_loader: - tensor 112: llama_model_loader: - tensor 113: llama_model_loader: - tensor 114: llama_model_loader: - tensor 115: llama_model_loader: - tensor 116: llama_model_loader: - tensor 117: llama_model_loader: - tensor 118: llama_model_loader: - tensor 119: llama_model_loader: - tensor 120: llama_model_loader: - tensor 121: llama_model_loader: - tensor 122: llama_model_loader: - tensor 123: llama_model_loader: - tensor 124: llama_model_loader: - tensor 125: llama_model_loader: - tensor 126: llama_model_loader: - tensor 127: llama_model_loader: - tensor 128: llama_model_loader: - tensor 129: llama_model_loader: - tensor 130: llama_model_loader: - tensor 131: llama_model_loader: - tensor 132: llama_model_loader: - tensor 133: llama_model_loader: - tensor 134: llama_model_loader: - tensor 135: llama_model_loader: - tensor 136: llama_model_loader: - tensor 137: llama_model_loader: - tensor 138: llama_model_loader: - tensor 139: llama_model_loader: - tensor 140: llama_model_loader: - tensor 141: llama_model_loader: - tensor 142: llama_model_loader: - tensor 143: llama_model_loader: - tensor 144: llama_model_loader: - tensor 145: llama_model_loader: - tensor 146: llama_model_loader: - tensor 147: llama_model_loader: - tensor 148: llama_model_loader: - tensor 149: llama_model_loader: - tensor 150: llama_model_loader: - tensor 151: llama_model_loader: - tensor 152: llama_model_loader: - tensor 153: llama_model_loader: - tensor 154: llama_model_loader: - tensor 155: llama_model_loader: - tensor 156: llama_model_loader: - tensor 157: llama_model_loader: - tensor 158: llama_model_loader: - tensor 159: llama_model_loader: - tensor 160: llama_model_loader: - tensor 161: llama_model_loader: - tensor 162: llama_model_loader: - tensor 163: llama_model_loader: - tensor 164: llama_model_loader: - tensor 165: llama_model_loader: - tensor 166: llama_model_loader: - tensor 167: llama_model_loader: - tensor 168: llama_model_loader: - tensor 169: llama_model_loader: - tensor 170: llama_model_loader: - tensor 171: llama_model_loader: - tensor 172: llama_model_loader: - tensor 173: llama_model_loader: - tensor 174: llama_model_loader: - tensor 175: llama_model_loader: - tensor 176: llama_model_loader: - tensor 177: llama_model_loader: - tensor 178: llama_model_loader: - tensor 179: llama_model_loader: - tensor 180: llama_model_loader: - tensor 181: llama_model_loader: - tensor 182: llama_model_loader: - tensor 183: llama_model_loader: - tensor 184: llama_model_loader: - tensor 185: llama_model_loader: - tensor 186: llama_model_loader: - tensor 187: llama_model_loader: - tensor 188: llama_model_loader: - tensor 189: llama_model_loader: - tensor 190: llama_model_loader: - tensor 191: llama_model_loader: - tensor 192: llama_model_loader: - tensor 193: llama_model_loader: - tensor 194: llama_model_loader: - tensor 195: llama_model_loader: - tensor 196: llama_model_loader: - tensor 197: llama_model_loader: - tensor 198: llama_model_loader: - tensor 199: llama_model_loader: - tensor 200: llama_model_loader: - tensor 201: llama_model_loader: - tensor 202: llama_model_loader: - tensor 203: llama_model_loader: - tensor 204: llama_model_loader: - tensor 205: llama_model_loader: - tensor 206: llama_model_loader: - tensor 207: llama_model_loader: - tensor 208: llama_model_loader: - tensor 209: llama_model_loader: - tensor 210: llama_model_loader: - tensor 211: llama_model_loader: - tensor 212: llama_model_loader: - tensor 213: llama_model_loader: - tensor 214: llama_model_loader: - tensor 215: llama_model_loader: - tensor 216: llama_model_loader: - tensor 217: llama_model_loader: - tensor 218: llama_model_loader: - tensor 219: llama_model_loader: - tensor 220: llama_model_loader: - tensor 221: llama_model_loader: - tensor 222: llama_model_loader: - tensor 223: llama_model_loader: - tensor 224: llama_model_loader: - tensor 225: llama_model_loader: - tensor 226: llama_model_loader: - tensor 227: llama_model_loader: - tensor 228: llama_model_loader: - tensor 229: llama_model_loader: - tensor 230: llama_model_loader: - tensor 231: llama_model_loader: - tensor 232: llama_model_loader: - tensor 233: llama_model_loader: - tensor 234: llama_model_loader: - tensor 235: llama_model_loader: - tensor 236: llama_model_loader: - tensor 237: llama_model_loader: - tensor 238: llama_model_loader: - tensor 239: llama_model_loader: - tensor 240: llama_model_loader: - tensor 241: llama_model_loader: - tensor 242: llama_model_loader: - tensor 243: llama_model_loader: - tensor 244: llama_model_loader: - tensor 245: llama_model_loader: - tensor 246: llama_model_loader: - tensor 247: llama_model_loader: - tensor 248: llama_model_loader: - tensor 249: llama_model_loader: - tensor 250: llama_model_loader: - tensor 251: llama_model_loader: - tensor 252: llama_model_loader: - tensor 253: llama_model_loader: - tensor 254: llama_model_loader: - tensor 255: llama_model_loader: - tensor 256: llama_model_loader: - tensor 257: llama_model_loader: - tensor 258: llama_model_loader: - tensor 259: llama_model_loader: - tensor 260: llama_model_loader: - tensor 261: llama_model_loader: - tensor 262: llama_model_loader: - tensor 263: llama_model_loader: - tensor 264: llama_model_loader: - tensor 265: llama_model_loader: - tensor 266: llama_model_loader: - tensor 267: llama_model_loader: - tensor 268: llama_model_loader: - tensor 269: llama_model_loader: - tensor 270: llama_model_loader: - tensor 271: llama_model_loader: - tensor 272: llama_model_loader: - tensor 273: llama_model_loader: - tensor 274: llama_model_loader: - tensor 275: llama_model_loader: - tensor 276: llama_model_loader: - tensor 277: llama_model_loader: - tensor 278: llama_model_loader: - tensor 279: llama_model_loader: - tensor 280: llama_model_loader: - tensor 281: llama_model_loader: - tensor 282: llama_model_loader: - tensor 283: llama_model_loader: - tensor 284: llama_model_loader: - tensor 285: llama_model_loader: - tensor 286: llama_model_loader: - tensor 287: llama_model_loader: - tensor 288: llama_model_loader: - tensor 289: llama_model_loader: - tensor 290: llama_model_loader: - tensor 291: llama_model_loader: - tensor 292: llama_model_loader: - tensor 293: llama_model_loader: - tensor 294: llama_model_loader: - tensor 295: llama_model_loader: - tensor 296: llama_model_loader: - tensor 297: llama_model_loader: - tensor 298: llama_model_loader: - tensor 299: llama_model_loader: - tensor 300: llama_model_loader: - tensor 301: llama_model_loader: - tensor 302: llama_model_loader: - tensor 303: llama_model_loader: - tensor 304: llama_model_loader: - tensor 305: llama_model_loader: - tensor 306: llama_model_loader: - tensor 307: llama_model_loader: - tensor 308: llama_model_loader: - tensor 309: llama_model_loader: - tensor 310: llama_model_loader: - tensor 311: llama_model_loader: - tensor 312: llama_model_loader: - tensor 313: llama_model_loader: - tensor 314: llama_model_loader: - tensor 315: llama_model_loader: - tensor 316: llama_model_loader: - tensor 317: llama_model_loader: - tensor 318: llama_model_loader: - tensor 319: llama_model_loader: - tensor 320: llama_model_loader: - tensor 321: llama_model_loader: - tensor 322: llama_model_loader: - tensor 323: llama_model_loader: - tensor 324: llama_model_loader: - tensor 325: llama_model_loader: - tensor 326: llama_model_loader: - tensor 327: llama_model_loader: - tensor 328: llama_model_loader: - tensor 329: llama_model_loader: - tensor 330: llama_model_loader: - tensor 331: llama_model_loader: - tensor 332: llama_model_loader: - tensor 333: llama_model_loader: - tensor 334: llama_model_loader: - tensor 335: llama_model_loader: - tensor 336: llama_model_loader: - tensor 337: llama_model_loader: - tensor 338: llama_model_loader: - tensor 339: llama_model_loader: - tensor 340: llama_model_loader: - tensor 341: llama_model_loader: - tensor 342: llama_model_loader: - tensor 343: llama_model_loader: - tensor 344: llama_model_loader: - tensor 345: llama_model_loader: - tensor 346: llama_model_loader: - tensor 347: llama_model_loader: - tensor 348: llama_model_loader: - tensor 349: llama_model_loader: - tensor 350: llama_model_loader: - tensor 351: llama_model_loader: - tensor 352: llama_model_loader: - tensor 353: llama_model_loader: - tensor 354: llama_model_loader: - tensor 355: llama_model_loader: - tensor 356: llama_model_loader: - tensor 357: llama_model_loader: - tensor 358: llama_model_loader: - tensor 359: llama_model_loader: - tensor 360: llama_model_loader: - tensor 361: llama_model_loader: - tensor 362: llama_model_loader: - tensor 363: llama_model_loader: - tensor 364: llama_model_loader: - tensor 365: llama_model_loader: - tensor 366: llama_model_loader: - tensor 367: llama_model_loader: - tensor 368: llama_model_loader: - tensor 369: llama_model_loader: - tensor 370: llama_model_loader: - tensor 371: llama_model_loader: - tensor 372: llama_model_loader: - tensor 373: llama_model_loader: - tensor 374: llama_model_loader: - tensor 375: llama_model_loader: - tensor 376: llama_model_loader: - tensor 377: llama_model_loader: - tensor 378: llama_model_loader: - tensor 379: llama_model_loader: - tensor 380: llama_model_loader: - tensor 381: llama_model_loader: - tensor 382: llama_model_loader: - tensor 383: llama_model_loader: - tensor 384: llama_model_loader: - tensor 385: llama_model_loader: - tensor 386: llama_model_loader: - tensor 387: llama_model_loader: - tensor 388: llama_model_loader: - tensor 389: llama_model_loader: - tensor 390: llama_model_loader: - tensor 391: llama_model_loader: - tensor 392: llama_model_loader: - tensor 393: llama_model_loader: - tensor 394: llama_model_loader: - tensor 395: llama_model_loader: - tensor 396: llama_model_loader: - tensor 397: llama_model_loader: - tensor 398: llama_model_loader: - tensor 399: llama_model_loader: - tensor 400: llama_model_loader: - tensor 401: llama_model_loader: - tensor 402: llama_model_loader: - tensor 403: llama_model_loader: - tensor 404: llama_model_loader: - tensor 405: llama_model_loader: - tensor 406: llama_model_loader: - tensor 407: llama_model_loader: - tensor 408: llama_model_loader: - tensor 409: llama_model_loader: - tensor 410: llama_model_loader: - tensor 411: llama_model_loader: - tensor 412: llama_model_loader: - tensor 413: llama_model_loader: - tensor 414: llama_model_loader: - tensor 415: llama_model_loader: - tensor 416: llama_model_loader: - tensor 417: llama_model_loader: - tensor 418: llama_model_loader: - tensor 419: llama_model_loader: - tensor 420: llama_model_loader: - tensor 421: llama_model_loader: - tensor 422: llama_model_loader: - tensor 423: llama_model_loader: - tensor 424: llama_model_loader: - tensor 425: llama_model_loader: - tensor 426: llama_model_loader: - tensor 427: llama_model_loader: - tensor 428: llama_model_loader: - tensor 429: llama_model_loader: - tensor 430: llama_model_loader: - tensor 431: llama_model_loader: - tensor 432: llama_model_loader: - tensor 433: llama_model_loader: - tensor 434: llama_model_loader: - tensor 435: llama_model_loader: - tensor 436: llama_model_loader: - tensor 437: llama_model_loader: - tensor 438: llama_model_loader: - tensor 439: llama_model_loader: - tensor 440: llama_model_loader: - tensor 441: llama_model_loader: - tensor 442: llama_model_loader: - tensor 443: llama_model_loader: - tensor 444: llama_model_loader: - tensor 445: llama_model_loader: - tensor 446: llama_model_loader: - tensor 447: llama_model_loader: - tensor 448: llama_model_loader: - tensor 449: llama_model_loader: - tensor 450: llama_model_loader: - tensor 451: llama_model_loader: - tensor 452: llama_model_loader: - tensor 453: llama_model_loader: - tensor 454: llama_model_loader: - tensor 455: llama_model_loader: - tensor 456: llama_model_loader: - tensor 457: llama_model_loader: - tensor 458: llama_model_loader: - tensor 459: llama_model_loader: - tensor 460: llama_model_loader: - tensor 461: llama_model_loader: - tensor 462: llama_model_loader: - tensor 463: llama_model_loader: - tensor 464: llama_model_loader: - tensor 465: llama_model_loader: - tensor 466: llama_model_loader: - tensor 467: llama_model_loader: - tensor 468: llama_model_loader: - tensor 469: llama_model_loader: - tensor 470: llama_model_loader: - tensor 471: llama_model_loader: - tensor 472: llama_model_loader: - tensor 473: llama_model_loader: - tensor 474: llama_model_loader: - tensor 475: llama_model_loader: - tensor 476: llama_model_loader: - tensor 477: llama_model_loader: - tensor 478: llama_model_loader: - tensor 479: llama_model_loader: - tensor 480: llama_model_loader: - tensor 481: llama_model_loader: - tensor 482: llama_model_loader: - tensor 483: llama_model_loader: - tensor 484: llama_model_loader: - tensor 485: llama_model_loader: - tensor 486: llama_model_loader: - tensor 487: llama_model_loader: - tensor 488: llama_model_loader: - tensor 489: llama_model_loader: - tensor 490: llama_model_loader: - tensor 491: llama_model_loader: - tensor 492: llama_model_loader: - tensor 493: llama_model_loader: - tensor 494: llama_model_loader: - tensor 495: llama_model_loader: - tensor 496: llama_model_loader: - tensor 497: llama_model_loader: - tensor 498: llama_model_loader: - tensor 499: llama_model_loader: - tensor 500: llama_model_loader: - tensor 501: llama_model_loader: - tensor 502: llama_model_loader: - tensor 503: llama_model_loader: - tensor 504: llama_model_loader: - tensor 505: llama_model_loader: - tensor 506: llama_model_loader: - tensor 507: llama_model_loader: - tensor 508: llama_model_loader: - tensor 509: llama_model_loader: - tensor 510: llama_model_loader: - tensor 511: llama_model_loader: - tensor 512: llama_model_loader: - tensor 513: llama_model_loader: - tensor 514: llama_model_loader: - tensor 515: llama_model_loader: - tensor 516: llama_model_loader: - tensor 517: llama_model_loader: - tensor 518: llama_model_loader: - tensor 519: llama_model_loader: - tensor 520: llama_model_loader: - tensor 521: llama_model_loader: - tensor 522: llama_model_loader: - tensor 523: llama_model_loader: - tensor 524: llama_model_loader: - tensor 525: llama_model_loader: - tensor 526: llama_model_loader: - tensor 527: llama_model_loader: - tensor 528: llama_model_loader: - tensor 529: llama_model_loader: - tensor 530: llama_model_loader: - tensor 531: llama_model_loader: - tensor 532: llama_model_loader: - tensor 533: llama_model_loader: - tensor 534: llama_model_loader: - tensor 535: llama_model_loader: - tensor 536: llama_model_loader: - tensor 537: llama_model_loader: - tensor 538: llama_model_loader: - tensor 539: llama_model_loader: - tensor 540: llama_model_loader: - tensor 541: llama_model_loader: - tensor 542: llama_model_loader: - tensor 543: llama_model_loader: - tensor 544: llama_model_loader: - tensor 545: llama_model_loader: - tensor 546: llama_model_loader: - tensor 547: llama_model_loader: - tensor 548: llama_model_loader: - tensor 549: llama_model_loader: - tensor 550: llama_model_loader: - tensor 551: llama_model_loader: - tensor 552: llama_model_loader: - tensor 553: llama_model_loader: - tensor 554: llama_model_loader: - tensor 555: llama_model_loader: - tensor 556: llama_model_loader: - tensor 557: llama_model_loader: - tensor 558: llama_model_loader: - tensor 559: llama_model_loader: - tensor 560: llama_model_loader: - tensor 561: llama_model_loader: - tensor 562: llama_model_loader: - tensor 563: llama_model_loader: - tensor 564: llama_model_loader: - tensor 565: llama_model_loader: - tensor 566: llama_model_loader: - tensor 567: llama_model_loader: - tensor 568: llama_model_loader: - tensor 569: llama_model_loader: - tensor 570: llama_model_loader: - tensor 571: llama_model_loader: - tensor 572: llama_model_loader: - tensor 573: llama_model_loader: - tensor 574: llama_model_loader: - tensor 575: llama_model_loader: - tensor 576: llama_model_loader: - tensor 577: llama_model_loader: - tensor 578: llama_model_loader: - tensor 579: llama_model_loader: - tensor 580: llama_model_loader: - tensor 581: llama_model_loader: - tensor 582: llama_model_loader: - tensor 583: llama_model_loader: - tensor 584: llama_model_loader: - tensor 585: llama_model_loader: - tensor 586: llama_model_loader: - tensor 587: llama_model_loader: - tensor 588: llama_model_loader: - tensor 589: llama_model_loader: - tensor 590: llama_model_loader: - tensor 591: llama_model_loader: - tensor 592: llama_model_loader: - tensor 593: llama_model_loader: - tensor 594: llama_model_loader: - tensor 595: llama_model_loader: - tensor 596: llama_model_loader: - tensor 597: llama_model_loader: - tensor 598: llama_model_loader: - tensor 599: llama_model_loader: - tensor 600: llama_model_loader: - tensor 601: llama_model_loader: - tensor 602: llama_model_loader: - tensor 603: llama_model_loader: - tensor 604: llama_model_loader: - tensor 605: llama_model_loader: - tensor 606: llama_model_loader: - tensor 607: llama_model_loader: - tensor 608: llama_model_loader: - tensor 609: llama_model_loader: - tensor 610: llama_model_loader: - tensor 611: llama_model_loader: - tensor 612: llama_model_loader: - tensor 613: llama_model_loader: - tensor 614: llama_model_loader: - tensor 615: llama_model_loader: - tensor 616: llama_model_loader: - tensor 617: llama_model_loader: - tensor 618: llama_model_loader: - tensor 619: llama_model_loader: - tensor 620: llama_model_loader: - tensor 621: llama_model_loader: - tensor 622: llama_model_loader: - tensor 623: llama_model_loader: - tensor 624: llama_model_loader: - tensor 625: llama_model_loader: - tensor 626: llama_model_loader: - tensor 627: llama_model_loader: - tensor 628: llama_model_loader: - tensor 629: llama_model_loader: - tensor 630: llama_model_loader: - tensor 631: llama_model_loader: - tensor 632: llama_model_loader: - tensor 633: llama_model_loader: - tensor 634: llama_model_loader: - tensor 635: llama_model_loader: - tensor 636: llama_model_loader: - tensor 637: llama_model_loader: - tensor 638: llama_model_loader: - tensor 639: llama_model_loader: - tensor 640: llama_model_loader: - tensor 641: llama_model_loader: - tensor 642: llama_model_loader: - tensor 643: llama_model_loader: - tensor 644: llama_model_loader: - tensor 645: llama_model_loader: - tensor 646: llama_model_loader: - tensor 647: llama_model_loader: - tensor 648: llama_model_loader: - tensor 649: llama_model_loader: - tensor 650: llama_model_loader: - tensor 651: llama_model_loader: - tensor 652: llama_model_loader: - tensor 653: llama_model_loader: - tensor 654: llama_model_loader: - tensor 655: llama_model_loader: - tensor 656: llama_model_loader: - tensor 657: llama_model_loader: - tensor 658: llama_model_loader: - tensor 659: llama_model_loader: - tensor 660: llama_model_loader: - tensor 661: llama_model_loader: - tensor 662: llama_model_loader: - tensor 663: llama_model_loader: - tensor 664: llama_model_loader: - tensor 665: llama_model_loader: - tensor 666: llama_model_loader: - tensor 667: llama_model_loader: - tensor 668: llama_model_loader: - tensor 669: llama_model_loader: - tensor 670: llama_model_loader: - tensor 671: llama_model_loader: - tensor 672: llama_model_loader: - tensor 673: llama_model_loader: - tensor 674: llama_model_loader: - tensor 675: llama_model_loader: - tensor 676: llama_model_loader: - tensor 677: llama_model_loader: - tensor 678: llama_model_loader: - tensor 679: llama_model_loader: - tensor 680: llama_model_loader: - tensor 681: llama_model_loader: - tensor 682: llama_model_loader: - tensor 683: llama_model_loader: - tensor 684: llama_model_loader: - tensor 685: llama_model_loader: - tensor 686: llama_model_loader: - tensor 687: llama_model_loader: - tensor 688: llama_model_loader: - tensor 689: llama_model_loader: - tensor 690: llama_model_loader: - tensor 691: llama_model_loader: - tensor 692: llama_model_loader: - tensor 693: llama_model_loader: - tensor 694: llama_model_loader: - tensor 695: llama_model_loader: - tensor 696: llama_model_loader: - tensor 697: llama_model_loader: - tensor 698: llama_model_loader: - tensor 699: llama_model_loader: - tensor 700: llama_model_loader: - tensor 701: llama_model_loader: - tensor 702: llama_model_loader: - tensor 703: llama_model_loader: - tensor 704: llama_model_loader: - tensor 705: llama_model_loader: - tensor 706: llama_model_loader: - tensor 707: llama_model_loader: - tensor 708: llama_model_loader: - tensor 709: llama_model_loader: - tensor 710: llama_model_loader: - tensor 711: llama_model_loader: - tensor 712: llama_model_loader: - tensor 713: llama_model_loader: - tensor 714: llama_model_loader: - tensor 715: llama_model_loader: - tensor 716: llama_model_loader: - tensor 717: llama_model_loader: - tensor 718: llama_model_loader: - tensor 719: llama_model_loader: - tensor 720: llama_model_loader: - tensor 721: llama_model_loader: - tensor 722: llama_model_loader: - kv 0: llama_model_loader: - kv 1: llama_model_loader: - kv 2: llama_model_loader: - kv 3: llama_model_loader: - kv 4: llama_model_loader: - kv 5: llama_model_loader: - kv 6: llama_model_loader: - kv 7: llama_model_loader: - kv 8: llama_model_loader: - kv 9: llama_model_loader: - kv 10: llama_model_loader: - kv 11: llama_model_loader: - kv 12: llama_model_loader: - kv 13: llama_model_loader: - kv 14: llama_model_loader: - kv 15: llama_model_loader: - type llama_model_loader: - type q8_0: llm_load_vocab: special tokens llm_load_print_meta: format llm_load_print_meta: arch llm_load_print_meta: vocab type llm_load_print_meta: n_vocab llm_load_print_meta: n_merges llm_load_print_meta: n_ctx_train llm_load_print_meta: n_embd llm_load_print_meta: n_head llm_load_print_meta: n_head_kv llm_load_print_meta: n_layer llm_load_print_meta: n_rot llm_load_print_meta: n_gqa llm_load_print_meta: f_norm_eps llm_load_print_meta: f_norm_rms_eps llm_load_print_meta: f_clamp_kqv llm_load_print_meta: f_max_alibi_bias llm_load_print_meta: n_ff llm_load_print_meta: freq_base_train llm_load_print_meta: freq_scale_train llm_load_print_meta: model type llm_load_print_meta: model ftype llm_load_print_meta: model params llm_load_print_meta: model size llm_load_print_meta: general.name llm_load_print_meta: BOS llm_load_print_meta: EOS llm_load_print_meta: UNK llm_load_print_meta: LF token llm_load_tensors: ggml ctx size = llm_load_tensors: using CUDA ggml_cuda_set_main_device: llm_load_tensors: mem required llm_load_tensors: offloading llm_load_tensors: offloaded llm_load_tensors: VRAM used: .................................. llama_new_context_with_model: n_ctx llama_new_context_with_model: freq_base llama_new_context_with_model: llama_new_context_with_model: kv self size llama_new_context_with_model: llama_new_context_with_model: llama_new_context_with_model: main: init model
print_params: n_vocab: 32000
print_params: n_ctx: 32
print_params: n_embd: 8192
print_params: n_ff: 28672
print_params: n_head: 64
print_params: n_head_kv: 8
print_params: n_layer: 80
print_params: norm_rms_eps print_params: rope_freq_base print_params: rope_freq_scale print_lora_params: n_rank_attention_norm print_lora_params: n_rank_wq print_lora_params: n_rank_wk print_lora_params: n_rank_wv print_lora_params: n_rank_wo print_lora_params: n_rank_ffn_norm print_lora_params: n_rank_w1 print_lora_params: n_rank_w2 print_lora_params: n_rank_w3 print_lora_params: n_rank_tok_embeddings print_lora_params: n_rank_norm print_lora_params: n_rank_output main: total train_iterations 0
main: seen train_samples 0
main: seen train_tokens 0
main: completed train_epochs 0
main: lora_size = 428339424 main: opt_size = 640969696 main: opt iter 0
main: input_size = 8192288 main: compute_size = 278948348096 main: evaluation order = terminate called after throwing what(): std::bad_alloc --model-base /srv/shared/llama.cpp/models/llama-2/llama-2-70b/ggml-model-q8_0.gguf --checkpoint-in llama-2-70b-finetune-data-LATEST.gguf --checkpoint-out llama-2-70b-finetune-data-ITERATION.gguf --lora-out llama-2-70b-finetune-data-ITERATION.bin --train-data /srv/shared/data.txt --save-every 50 --threads 48 --batch 2 --grad-acc 2 --adam-alpha 0.0003 --ctx 32 --sample-start '### Instruction:' --include-sample-start --no-checkpointing
ama.cpp/models/llama-2/llama-2-70b/ggml-model-q8_0.gguf'
no
yes
CUDA devices:
capability 8.9
capability 8.9
capability 8.9
capability 8.9
meta data with 16 key-value pairs and 723 tensors from /srv/shared/llama.cpp/models/llama-2/llama-2-70b/ggml-model-q8_0.gguf (version GGUF V3 (latest))
token_embd.weight q8_0 [ 8192, 32000, 1, 1 ]
output_norm.weight f32 [ 8192, 1, 1, 1 ]
output.weight q8_0 [ 8192, 32000, 1, 1 ]
blk.0.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.0.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.0.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.0.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.0.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.0.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.0.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.0.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.0.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.1.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.1.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.1.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.1.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.1.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.1.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.1.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.1.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.1.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.2.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.2.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.2.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.2.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.2.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.2.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.2.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.2.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.2.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.3.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.3.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.3.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.3.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.3.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.3.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.3.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.3.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.3.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.4.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.4.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.4.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.4.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.4.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.4.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.4.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.4.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.4.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.5.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.5.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.5.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.5.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.5.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.5.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.5.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.5.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.5.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.6.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.6.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.6.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.6.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.6.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.6.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.6.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.6.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.6.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.7.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.7.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.7.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.7.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.7.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.7.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.7.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.7.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.7.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.8.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.8.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.8.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.8.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.8.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.8.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.8.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.8.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.8.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.9.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.9.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.9.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.9.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.9.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.9.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.9.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.9.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.9.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.10.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.10.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.10.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.10.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.10.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.10.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.10.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.10.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.10.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.11.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.11.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.11.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.11.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.11.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.11.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.11.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.11.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.11.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.12.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.12.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.12.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.12.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.12.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.12.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.12.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.12.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.12.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.13.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.13.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.13.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.13.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.13.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.13.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.13.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.13.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.13.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.14.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.14.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.14.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.14.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.14.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.14.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.14.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.14.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.14.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.15.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.15.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.15.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.15.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.15.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.15.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.15.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.15.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.15.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.16.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.16.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.16.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.16.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.16.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.16.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.16.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.16.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.16.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.17.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.17.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.17.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.17.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.17.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.17.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.17.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.17.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.17.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.18.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.18.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.18.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.18.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.18.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.18.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.18.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.18.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.18.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.19.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.19.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.19.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.19.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.19.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.19.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.19.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.19.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.19.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.20.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.20.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.20.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.20.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.20.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.20.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.20.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.20.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.20.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.21.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.21.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.21.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.21.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.21.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.21.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.21.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.21.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.21.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.22.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.22.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.22.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.22.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.22.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.22.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.22.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.22.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.22.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.23.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.23.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.23.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.23.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.23.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.23.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.23.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.23.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.23.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.24.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.24.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.24.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.24.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.24.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.24.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.24.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.24.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.24.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.25.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.25.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.25.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.25.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.25.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.25.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.25.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.25.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.25.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.26.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.26.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.26.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.26.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.26.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.26.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.26.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.26.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.26.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.27.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.27.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.27.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.27.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.27.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.27.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.27.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.27.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.27.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.28.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.28.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.28.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.28.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.28.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.28.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.28.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.28.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.28.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.29.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.29.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.29.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.29.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.29.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.29.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.29.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.29.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.29.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.30.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.30.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.30.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.30.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.30.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.30.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.30.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.30.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.30.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.31.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.31.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.31.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.31.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.31.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.31.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.31.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.31.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.31.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.32.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.32.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.32.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.32.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.32.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.32.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.32.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.32.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.32.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.33.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.33.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.33.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.33.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.33.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.33.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.33.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.33.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.33.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.34.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.34.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.34.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.34.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.34.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.34.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.34.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.34.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.34.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.35.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.35.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.35.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.35.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.35.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.35.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.35.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.35.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.35.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.36.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.36.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.36.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.36.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.36.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.36.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.36.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.36.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.36.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.37.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.37.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.37.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.37.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.37.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.37.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.37.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.37.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.37.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.38.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.38.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.38.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.38.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.38.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.38.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.38.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.38.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.38.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.39.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.39.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.39.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.39.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.39.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.39.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.39.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.39.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.39.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.40.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.40.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.40.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.40.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.40.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.40.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.40.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.40.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.40.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.41.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.41.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.41.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.41.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.41.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.41.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.41.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.41.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.41.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.42.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.42.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.42.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.42.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.42.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.42.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.42.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.42.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.42.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.43.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.43.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.43.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.43.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.43.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.43.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.43.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.43.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.43.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.44.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.44.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.44.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.44.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.44.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.44.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.44.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.44.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.44.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.45.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.45.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.45.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.45.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.45.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.45.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.45.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.45.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.45.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.46.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.46.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.46.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.46.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.46.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.46.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.46.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.46.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.46.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.47.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.47.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.47.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.47.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.47.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.47.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.47.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.47.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.47.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.48.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.48.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.48.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.48.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.48.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.48.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.48.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.48.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.48.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.49.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.49.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.49.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.49.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.49.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.49.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.49.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.49.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.49.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.50.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.50.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.50.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.50.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.50.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.50.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.50.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.50.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.50.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.51.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.51.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.51.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.51.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.51.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.51.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.51.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.51.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.51.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.52.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.52.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.52.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.52.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.52.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.52.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.52.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.52.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.52.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.53.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.53.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.53.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.53.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.53.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.53.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.53.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.53.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.53.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.54.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.54.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.54.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.54.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.54.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.54.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.54.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.54.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.54.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.55.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.55.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.55.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.55.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.55.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.55.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.55.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.55.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.55.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.56.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.56.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.56.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.56.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.56.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.56.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.56.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.56.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.56.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.57.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.57.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.57.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.57.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.57.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.57.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.57.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.57.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.57.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.58.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.58.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.58.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.58.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.58.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.58.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.58.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.58.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.58.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.59.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.59.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.59.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.59.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.59.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.59.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.59.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.59.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.59.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.60.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.60.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.60.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.60.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.60.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.60.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.60.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.60.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.60.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.61.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.61.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.61.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.61.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.61.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.61.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.61.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.61.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.61.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.62.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.62.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.62.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.62.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.62.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.62.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.62.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.62.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.62.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.63.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.63.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.63.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.63.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.63.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.63.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.63.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.63.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.63.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.64.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.64.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.64.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.64.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.64.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.64.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.64.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.64.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.64.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.65.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.65.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.65.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.65.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.65.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.65.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.65.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.65.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.65.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.66.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.66.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.66.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.66.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.66.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.66.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.66.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.66.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.66.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.67.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.67.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.67.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.67.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.67.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.67.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.67.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.67.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.67.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.68.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.68.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.68.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.68.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.68.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.68.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.68.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.68.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.68.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.69.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.69.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.69.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.69.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.69.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.69.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.69.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.69.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.69.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.70.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.70.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.70.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.70.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.70.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.70.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.70.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.70.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.70.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.71.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.71.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.71.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.71.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.71.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.71.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.71.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.71.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.71.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.72.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.72.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.72.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.72.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.72.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.72.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.72.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.72.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.72.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.73.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.73.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.73.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.73.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.73.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.73.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.73.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.73.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.73.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.74.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.74.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.74.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.74.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.74.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.74.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.74.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.74.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.74.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.75.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.75.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.75.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.75.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.75.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.75.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.75.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.75.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.75.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.76.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.76.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.76.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.76.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.76.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.76.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.76.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.76.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.76.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.77.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.77.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.77.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.77.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.77.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.77.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.77.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.77.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.77.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.78.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.78.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.78.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.78.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.78.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.78.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.78.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.78.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.78.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.79.attn_q.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.79.attn_k.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.79.attn_v.weight q8_0 [ 8192, 1024, 1, 1 ]
blk.79.attn_output.weight q8_0 [ 8192, 8192, 1, 1 ]
blk.79.ffn_gate.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.79.ffn_down.weight q8_0 [ 28672, 8192, 1, 1 ]
blk.79.ffn_up.weight q8_0 [ 8192, 28672, 1, 1 ]
blk.79.attn_norm.weight f32 [ 8192, 1, 1, 1 ]
blk.79.ffn_norm.weight f32 [ 8192, 1, 1, 1 ]
general.architecture str
general.name str
llama.context_length u32
llama.embedding_length u32
llama.block_count u32
llama.feed_forward_length u32
llama.rope.dimension_count u32
llama.attention.head_count u32
llama.attention.head_count_kv u32
llama.attention.layer_norm_rms_epsilon f32
general.file_type u32
tokenizer.ggml.model str
tokenizer.ggml.tokens arr
tokenizer.ggml.scores arr
tokenizer.ggml.token_type arr
general.quantization_version u32
f32: 161 tensors
562 tensors
definition check successful ( 259/32000 ).
= GGUF V3 (latest)
= llama
= SPM
= 32000
= 0
= 4096
= 8192
= 64
= 8
= 80
= 128
= 8
= 0.0e+00
= 1.0e-05
= 0.0e+00
= 0.0e+00
= 28672
= 10000.0
= 1
= 70B
= mostly Q8_0
= 68.98 B
= 68.26 GiB (8.50 BPW)
= LLaMA v2
token = 1 ''
token = 2 ''
token = 0 ''
= 13 '<0x0A>'
0.24 MB
for GPU acceleration
using device 0 (NVIDIA L4) as main device
= 69896.52 MB
0 repeating layers to GPU
0/83 layers to GPU
0.00 MB
..................................................................
= 512
= 10000.0
freq_scale = 1
= 160.00 MB
compute buffer total size = 151.13 MB
VRAM scratch buffer: 145.00 MB
total VRAM used: 145.00 MB (model: 0.00 MB, context: 145.00 MB)
: 0.000010
: 10000.000000
: 1.000000
: 1
: 4
: 4
: 4
: 4
: 1
: 4
: 4
: 4
: 4
: 1
: 4
bytes (408.5 MB)
bytes (611.3 MB)
bytes (7.8 MB)
bytes (266025.9 MB)
RIGHT_TO_LEFT
an instance of 'std::bad_alloc'

AndrewGodfrey · 2023-11-15T02:19:57Z

I'm guessing the call stack where it asserts is init_lora > alloc_lora > ggml_allocr_alloc.
Could you maybe confirm that, e.g. in a debugger (breakpoint on abort()) or by adding logging?

If so, it could mean the "// measure data size" code in init_lora is coming up with a value for 'size' that is too low.

github-actions · 2024-04-04T01:08:09Z

This issue was closed because it has been inactive for 14 days since being marked as stale.

Taikono-Himazin mentioned this issue Dec 6, 2023

When I try to do finetuning I get a GGML_ASSERT: ggml.c:16911: np < GGML_MAX_PARAMS error. #4342

Closed

4 tasks

github-actions bot added the stale label Mar 19, 2024

github-actions bot closed this as completed Apr 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[User] Unable to Finetune Llama 2 70B #3644

[User] Unable to Finetune Llama 2 70B #3644

mrroll commented Oct 16, 2023 •

edited

Loading

RedAndr commented Oct 16, 2023

mrroll commented Oct 16, 2023 •

edited

Loading

QueryType commented Oct 25, 2023 •

edited

Loading

KerfuffleV2 commented Oct 25, 2023

QueryType commented Oct 26, 2023

mrroll commented Oct 26, 2023

QueryType commented Oct 26, 2023

mrroll commented Oct 27, 2023

QueryType commented Oct 27, 2023

mrroll commented Oct 27, 2023

slaren commented Oct 27, 2023

mrroll commented Oct 30, 2023 •

edited

Loading

AndrewGodfrey commented Nov 15, 2023

github-actions bot commented Apr 4, 2024

[User] Unable to Finetune Llama 2 70B #3644

[User] Unable to Finetune Llama 2 70B #3644

Comments

mrroll commented Oct 16, 2023 • edited Loading

Prerequisites

Expected Behavior

Current Behavior

Environment and Context

Failure Information (for bugs)

Steps to Reproduce

Failure Logs

RedAndr commented Oct 16, 2023

mrroll commented Oct 16, 2023 • edited Loading

QueryType commented Oct 25, 2023 • edited Loading

KerfuffleV2 commented Oct 25, 2023

QueryType commented Oct 26, 2023

mrroll commented Oct 26, 2023

QueryType commented Oct 26, 2023

mrroll commented Oct 27, 2023

QueryType commented Oct 27, 2023

mrroll commented Oct 27, 2023

slaren commented Oct 27, 2023

mrroll commented Oct 30, 2023 • edited Loading

AndrewGodfrey commented Nov 15, 2023

github-actions bot commented Apr 4, 2024

mrroll commented Oct 16, 2023 •

edited

Loading

mrroll commented Oct 16, 2023 •

edited

Loading

QueryType commented Oct 25, 2023 •

edited

Loading

mrroll commented Oct 30, 2023 •

edited

Loading