[GGML] Added RISC-V Vector Intrinsics Support #2929
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi,
In this PR, we have added the RISC-V intrinsics for the following vector dot product functions
In future, this will enable GGML to run efficiently on RISC-V hardware with vector support and also open a way to compare its performance with other vector processors like Intel AVX and Arm Neon. This will also led to performance improvement and speedup for application using GGML on the RISC-V hardware with vector processor
The output is tested and verified for each of the legacy ggml 7B quantize models (includes q4_0, q4_1, q5_0, q5_1 and q8_0) by using qemu-riscv64 emulator.
Edit: LLaMa.cpp has stop using GGML format after 22nd August, 2023, and instead shifted to new format GGUF. So these functions will not be effective for LLaMa.cpp after this, soon will submit new PR with support for GGUF
[Cross Compiling Environment]
Ubuntu: 22.10
riscv-toolchain: 2023.07.05 riscv64 linux glibc
On the actual hardware there will be no issue but running with qemu require to sligthly modify the Makefile, we just have to overwrite the CC and CCX variable with toolchain gcc compiler and also specify the architecture flag for the Makefile
And then after it run make
[QEMU]
[Output]
If you'd like to test these changes, we've set up a cloud-v pipeline on our fork repository (main branch), which you can use to run and verify the code on riscv-qemu emulator
Any feedback is welcome, if you have any suggestions or improvements, please share.