There is a Quick Start on llama.cpp’s GitHub page but this page is focused on using containers.

The models are big so I prefer to download them once, not on every container run.

Download models

#!/bin/bash

declare -a HF_MODELS=(
    'google/gemma-4-12B-it-qat-q4_0-gguf'
    'google/gemma-4-E4B-it-qat-q4_0-gguf'
    'google/gemma-4-E2B-it-qat-q4_0-gguf'
    'unsloth/gemma-4-12B-it-qat-GGUF:UD-Q4_K_XL'
    'unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL'
    'unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_XL'
)

mkdir -p "$HOME"/llama/models

for model_name in "${HF_MODELS[@]}"; do
    podman run -it --rm \
        --volume $HOME/llama/models:/models \
        --env LLAMA_CACHE=/models \
        ghcr.io/ggml-org/llama.cpp:light -hf "${model_name}"
done

Benchmark

cd "$HOME/llama/models"

podman run -it --rm \
    --volume $HOME/llama/models:/models \
    --env LLAMA_CACHE=/models \
    ghcr.io/ggml-org/llama.cpp:full --bench \
    $(find . -name '*.gguf' -not -name '*mmproj*' -exec printf -- '-m /models/%s ' {} +) \
    -t 12 --progress

load_backend: loaded CPU backend from /app/libggml-cpu-alderlake.so

CPU vs GPU

podman run -it --rm \
    --volume $HOME/llama/models:/models --env LLAMA_CACHE=/models \
    ghcr.io/ggml-org/llama.cpp:full --bench \
    -m /models/models--google--gemma-4-E2B-it-qat-q4_0-gguf/snapshots/1894d1fc0a19d86697abd40483f5983c867df03f/gemma-4-E2B_q4_0-it.gguf \
    -t 12 --progress

load_backend: loaded CPU backend from /app/libggml-cpu-alderlake.so

I only have integrated GPU, nothing really interesting. Use ls -l /dev/dri to find the graphic cards. For intel:

podman run -it --rm \
    --device /dev/dri/renderD128:/dev/dri/renderD128 \
    --device /dev/dri/card1:/dev/dri/card1 \
    --volume $HOME/llama/models:/models --env LLAMA_CACHE=/models \
    ghcr.io/ggml-org/llama.cpp:full-intel --bench \
    -m /models/models--google--gemma-4-E2B-it-qat-q4_0-gguf/snapshots/1894d1fc0a19d86697abd40483f5983c867df03f/gemma-4-E2B_q4_0-it.gguf \
    -ngl -1 --progress

load_backend: loaded SYCL backend from /app/libggml-sycl.so load_backend: loaded CPU backend from /app/libggml-cpu-alderlake.so

Run

For tests

Run unsloth/gemma-4-26B-A4B-it-qat-GGUF with Unsloth’s recommended settings:

model='/models/models--unsloth--gemma-4-26B-A4B-it-qat-GGUF/snapshots/02749a7b272109255a4c559a80894d3d9777574c/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf'
podman run --rm -d --name llama-test \
    --volume "$HOME/llama/models":/models:ro --env LLAMA_CACHE=/models \
    -p 127.0.0.1:8080:8080 \
    ghcr.io/ggml-org/llama.cpp:server -m "$model" \
    --temp 1.0 --top-p 0.95 --top-k 64 \
    --host 0.0.0.0 --port 8080

then point your browser to http://127.0.0.1:8080

Production environment

For production, the container should be restricted. Example:

model='/models/models--unsloth--gemma-4-26B-A4B-it-qat-GGUF/snapshots/02749a7b272109255a4c559a80894d3d9777574c/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf'
podman run -d --name llama-production \
    --read-only --tmpfs /tmp \
    --security-opt no-new-privileges \
    --cap-drop=ALL \
    --volume "$HOME/llama/models":/models:ro --env LLAMA_CACHE=/models \
    -p 127.0.0.1:8080:8080 \
    ghcr.io/ggml-org/llama.cpp:server -m "$model" \
    --temp 1.0 --top-p 0.95 --top-k 64 \
    --host 0.0.0.0 --port 8080

then point your browser to http://127.0.0.1:8080

Also, -c and -n can be used to explicitly restrict de input context size and the maximum predicted tokens. Example: -c 4096 -n 512

Cleanup with:

podman stop llama-production
podman rm llama-production

Images cleanup

# display the images for current user
podman images

# remove **all** unused images for current user
podman image prune -a

Want to share this post? Click to copy the link.