Dockerfiles I'm testing with right now

[spirilis]

Just posting this as a WIP as I tinker more with this project... I unfortunately don't have a GPU to play with (using an old HP server pizzabox) but at least have the cuBLAS stuff building, and will test this at work on our GPU cloud instances at some point...

Getting the bleeding-edge version of llama-cpp-python in this one:

Dockerfile.githead-llama-cpp-python

FROM ubuntu:22.04 AS build

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update && apt-get -y install build-essential cmake git wget python3 python3-pip

RUN mkdir /build

WORKDIR /build

RUN cd /build && git clone https://github.com/abetlen/llama-cpp-python && \
    cd llama-cpp-python && \
    git submodule init && git submodule update && \
    pip3 install fastapi uvicorn sse-starlette scikit-build pathlib && \
    python3 setup.py install && \
    sed -i 's/host=os.getenv("HOST", "localhost")/host="0.0.0.0"/' /usr/local/lib/python3*/dist-packages/llama_cpp*.egg/llama_cpp/server/__main__.py

RUN cd /usr/local && tar zcf /build/built-llama-cpp-python.tar.gz lib/python* bin/uvicorn



FROM ubuntu:22.04 AS runtime

RUN apt-get update && apt-get -y install curl bash numactl wget less vim tmux python3

WORKDIR /app

COPY --from=build /build/built-llama-cpp-python.tar.gz /app

RUN cd /usr/local && tar zxvfp /app/built-llama-cpp-python.tar.gz && rm -f /app/built-llama-cpp-python.tar.gz

COPY entrypoint-python.sh /app/

RUN chmod 755 /app/entrypoint-python.sh

ENTRYPOINT ["/app/entrypoint-python.sh"]

The "entrypoint-python.sh" follows:

entrypoint-python.sh

#!/bin/sh

if [ "$1" = "/bin/sh" -o "$1" = "/bin/bash" ]; then
    exec $1
fi

SERVER_PRECMD=""
if [ -n "$NUMACTL_OPTS" ]; then
    SERVER_PRECMD="${SERVER_PRECMD} numactl ${NUMACTL_OPTS}"
fi

SERVER_ARGS=""

if [ -n "$MODEL" ]; then
    SERVER_ARGS="${SERVER_ARGS} --model ${MODEL}"
fi

if [ -n "$N_THREADS" ]; then
    SERVER_ARGS="${SERVER_ARGS} --n_threads ${N_THREADS}"
fi

if [ -n "$N_GPU_LAYERS" ]; then
    SERVER_ARGS="${SERVER_ARGS} --n_gpu_layers ${N_GPU_LAYERS}"
fi

if [ -n "$NGL" ]; then
    SERVER_ARGS="${SERVER_ARGS} --n_gpu_layers ${NGL}"
fi

if [ -n "$N_BATCH" ]; then
    SERVER_ARGS="${SERVER_ARGS} --n_batch ${N_BATCH}"
fi

${SERVER_PRECMD} python3 -m llama_cpp.server ${SERVER_ARGS}

(I'm not sure if I need to fan out those $MODEL, $N_THREADS, $N_GPU_LAYERS et al into CLI options - guess I should test that soon)

CUDA build in next post-

[spirilis]

This one builds, but I don't have the hardware to test it yet-

Dockerfile.githead-llama-cpp-python-cuBLAS

FROM ubuntu:22.04 AS build

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update && apt-get -y install build-essential cmake git wget python3 python3-pip

# cuBLAS installation
RUN wget -q https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin && \
    mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600 && \
    wget -q https://developer.download.nvidia.com/compute/cuda/12.1.1/local_installers/cuda-repo-ubuntu2204-12-1-local_12.1.1-530.30.02-1_amd64.deb && \
    dpkg -i cuda-repo-ubuntu2204-12-1-local_12.1.1-530.30.02-1_amd64.deb && \
    cp /var/cuda-repo-ubuntu2204-12-1-local/cuda-*-keyring.gpg /usr/share/keyrings/ && \
    apt-get update

RUN apt-get -y install libcublas-12-1 libcublas-dev-12-1 cuda-nvcc-12-1

ENV PATH "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/cuda-12.1/bin"

RUN mkdir /build

WORKDIR /build

RUN cd /build && git clone --recurse-submodules https://github.com/abetlen/llama-cpp-python && \
    cd llama-cpp-python && \
    pip3 install fastapi uvicorn sse-starlette scikit-build pathlib && \
    CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 python3 setup.py install && \
    sed -i 's/host=os.getenv("HOST", "localhost")/host="0.0.0.0"/' /usr/local/lib/python3*/dist-packages/llama_cpp*.egg/llama_cpp/server/__main__.py

RUN cd /usr/local && tar zcf /build/built-llama-cpp-python.tar.gz lib/python* bin/uvicorn



FROM ubuntu:22.04 AS runtime

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update && apt-get -y install curl bash numactl wget python3

# cuBLAS installation - runtime libraries for running llama.cpp
RUN wget -q https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin && \
    mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600 && \
    wget -q https://developer.download.nvidia.com/compute/cuda/12.1.1/local_installers/cuda-repo-ubuntu2204-12-1-local_12.1.1-530.30.02-1_amd64.deb && \
    dpkg -i cuda-repo-ubuntu2204-12-1-local_12.1.1-530.30.02-1_amd64.deb && \
    cp /var/cuda-repo-ubuntu2204-12-1-local/cuda-*-keyring.gpg /usr/share/keyrings/ && \
    apt-get update

RUN apt-get -y install cuda-cudart-12-1 libcublas-12-1 && \
    rm -f /cuda-repo-ubuntu2204-12-1-local* && rm -Rf /var/cuda-repo-ubuntu2204-12-1-local/

WORKDIR /app

COPY --from=build /build/built-llama-cpp-python.tar.gz /app

RUN cd /usr/local && tar zxvfp /app/built-llama-cpp-python.tar.gz && rm -f /app/built-llama-cpp-python.tar.gz

COPY entrypoint-python.sh /app/

RUN chmod 755 /app/entrypoint-python.sh

ENTRYPOINT ["/app/entrypoint-python.sh"]

[gjmulder]

There are already working server Docker files here that support both OpenBLAS and CuBLAS, and also a completely self-contained Open Llama open source Docker build that includes downloading and packaging of the model:

https://github.com/abetlen/llama-cpp-python/tree/main/docker