Ollama on Thumper

Ollama is a lightweight LLM inference server that wraps llama.cpp behind an OpenAI-compatible REST API. It handles model loading, GPU offloading, and concurrent requests — so individual apps don’t have to.

Five or more apps in the Thumper catalog depend on Ollama as their LLM backend:

• Open WebUI — full-featured chat interface with RAG, web search, and multi-model support
• Jan — native desktop LLM chat (can also use its built-in llama.cpp engine)
• SillyTavern — character-based roleplay and creative writing frontend
• Thumper Agent — the built-in autonomous agent for code, research, and automation tasks
• Custom apps — any app that targets the OpenAI-compatible API

Rather than bundling separate LLM engines in each app, Thumper runs a single Ollama instance that all apps share. This saves disk space, avoids duplicate model downloads, and prevents VRAM conflicts from multiple processes trying to load models simultaneously.

When you launch an app that needs Ollama, Thumper handles the full lifecycle automatically:

Auto-Start

If Ollama is not already running, Thumper starts it as a background process before launching the app. The app waits for the Ollama health check to pass before connecting.

Auto-Pull Models

Each app’s manifest declares which Ollama models it requires. On first launch, Thumper checks which models are missing and streams them automatically with progress reporting.

Port Override

Ollama defaults to port 11434. If that port is already in use (e.g., by a system-installed Ollama), Thumper can override it via the manifest’s ollama_port_override field. The port is dynamically substituted into app launch patches so the app connects to the correct instance.

# Default Ollama port
OLLAMA_HOST=http://localhost:11434
 
# Override example (set in .thumper.yaml)
ollama_port_override: 11435

Preferred Model Tags

Apps can specify an ollama_tag in their manifest to prefer a specific model variant. For example, an app might request qwen3:4b instead of the default tag. Thumper resolves the preferred tag and falls back to the pack’s default model if the preferred one is unavailable.

Ollama stores models in its own cache directory, separate from the HuggingFace cache. You can manage models directly through the Ollama CLI:

# List installed models
ollama list
 
# Pull a new model
ollama pull llama3.2
 
# Remove a model to free disk space
ollama rm codellama:13b
 
# Show model details (size, quantization, parameters)
ollama show llama3.2

Common Models

Custom Fine-Tuned Models

You can run your own fine-tuned models in Ollama using a Modelfile that references a GGUF file. See the LLM Fine-Tuning Guide for the full workflow: training with QLoRA, exporting to GGUF, and creating an Ollama Modelfile.

Ollama exposes a REST API on localhost:11434 (or the overridden port). The API is OpenAI-compatible, so any tool that works with the OpenAI API can point at Ollama instead.

Key Endpoints

Example Requests

# Chat completion
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [{"role": "user", "content": "Hello!"}],
  "stream": false
}'
 
# List installed models
curl http://localhost:11434/api/tags
 
# Single-turn generation
curl http://localhost:11434/api/generate -d '{
  "model": "qwen3:4b",
  "prompt": "Explain GGUF in one paragraph",
  "stream": false
}'