llama.cpp on Thumper

llama.cpp is a pure C/C++ implementation of LLM inference. It loads GGUF model files and runs them on CPU, GPU, or a mix of both. It supports dozens of model architectures (LLaMA, Mistral, Qwen, Phi, Gemma, and more) through a single unified runtime.

Thumper uses llama.cpp in three ways:

• Via Ollama — Ollama wraps llama.cpp behind an OpenAI-compatible API, serving models for Open WebUI, SillyTavern, and other API consumers
• Via Jan — Jan bundles llama.cpp as its native inference engine for standalone desktop chat
• Embedded — Thumper links llama.cpp directly via the Rust llama-cpp-2 crate for the built-in agent, prompt enhancement, and background tasks

The embedded path uses dynamic GPU backend loading (GGML_BACKEND_DL), which means GPU support is loaded at runtime rather than compiled in. This allows a single Thumper binary to work with CUDA, ROCm, Metal, Vulkan, or CPU — depending on what’s available on the system.

GGUF (GPT-Generated Unified Format) is the standard file format for llama.cpp models. A single .gguf file contains everything needed for inference:

• Model weights — the actual parameters, typically quantized to reduce size
• Tokenizer — vocabulary and merges, embedded in the file
• Metadata — architecture type, context length, quantization scheme, author, license
• Chat template — Jinja2 template for formatting messages (system, user, assistant roles)

The key advantage of GGUF is portability. Unlike safetensors or PyTorch checkpoints that require a separate tokenizer config, GGUF is fully self-contained. You can copy a single file to another machine and it just works.

Quantization reduces model size and memory usage by storing weights at lower precision. llama.cpp supports many quantization levels, each trading quality for speed and size.

Thumper uses llama.cpp’s dynamic backend loading system (GGML_BACKEND_DL). Instead of compiling separate binaries for each GPU vendor, a single binary loads the appropriate GPU backend at runtime by calling ggml_backend_load_all().

Supported Backends

The dynamic loading approach means Thumper works on any hardware out of the box. If CUDA libraries are found, the CUDA backend is loaded automatically. If not, it falls back to the next available option. No configuration needed.

Context length determines how much text the model can process at once. Longer context uses more memory and slows inference.

Sampling controls how the model picks the next token. These parameters affect creativity, coherence, and response diversity.

Temperature

Controls randomness. Lower values (0.1–0.3) produce focused, deterministic output. Higher values (0.7–1.0) produce more creative, varied output. Default: 0.8.

Top-p (Nucleus Sampling)

Keeps only tokens whose cumulative probability exceeds the threshold. A top_p of 0.9 means the model considers the smallest set of tokens covering 90% probability. Default: 0.9.

Top-k

Limits the model to the k most likely tokens at each step. Top_k of 40 means only the 40 most probable tokens are considered. Lower values = more focused. Default: 40.

Recommended Presets

• Code generation — temperature=0.2, top_p=0.9, top_k=20
• Chat / conversation — temperature=0.7, top_p=0.9, top_k=40
• Creative writing — temperature=1.0, top_p=0.95, top_k=60
• Factual Q&A — temperature=0.1, top_p=0.8, top_k=10