~8 min read

Chat with a Local AI

~10 minBeginnerYou'll build: A conversation with a local AI chatbot

In this tutorial you’ll install Open WebUI, pull a language model through Ollama, and have your first conversation with a locally-running AI chatbot — completely offline and private.

Step 1: Prerequisites

This tutorial works on any hardware — no GPU required. A dedicated GPU speeds up responses, but CPU-only mode works fine for getting started.

You’ll need Thumper-Run installed on your machine. If you haven’t done that yet, follow the installation steps in the First Image tutorial (Step 1 only) and come back here.

Thumper-Run installed and running
At least 8 GB of free disk space (for the app + default model)
An internet connection for the initial download (after that, everything runs offline)

Step 2: Install Open WebUI

Open WebUI is a polished chat interface that connects to Ollama for local language models. Thumper-Run bundles both together so you don’t need to configure anything manually.

In the catalog, find Open WebUI (or type "open webui" in the search bar)
Click the Install button on the app card
Watch the console as pip installs dependencies — this takes 2–5 minutes

[INFO] Creating virtual environment...
[INFO] Installing open-webui (pip)...
Collecting open-webui
  Downloading open_webui-0.6.x.tar.gz (42 MB)
Installing collected packages: open-webui
[INFO] Installation complete.

Test it: The app card should now show a Launch button instead of Install.

Make sure you have at least 8 GB of free disk space before installing. The Open WebUI Python environment plus the default model require roughly 6–7 GB total.

Step 3: Pull a Model

Ollama manages language models for you. When you launch Open WebUI for the first time, Thumper-Run starts Ollama in the background and pulls a default model automatically.

Click Launch on the Open WebUI card
Watch the console — Ollama starts first, then the model downloads
The default model (Llama 3.2 3B) is ~2 GB and takes 1–3 minutes to download
Once the model is ready, the status changes to "Running"

[INFO] Starting Ollama server...
[INFO] Ollama listening on 127.0.0.1:11434
[INFO] Pulling model: llama3.2:3b
pulling manifest...
pulling dde5aa3fc5ff... 100% 2.0 GB
verifying sha256 digest
writing manifest
[INFO] Model ready.

Test it: In the Thumper-Run dashboard, click the running app to open its detail view, then select the Console tab in the sidebar. You should see "Model ready" with no errors. Back in the Open WebUI chat interface, the chat input box becomes active (no longer grayed out) when the model is loaded and ready.

If you see "connection refused" errors, another process may already be using port 11434. Stop any existing Ollama instance before launching.

Ollama’s API is available at http://localhost:11434/api/chat for custom integrations. You can use it from scripts, notebooks, or other tools that speak the Ollama protocol.

Step 4: Chat

Your local chatbot is ready. Time to talk to it!

Type a message in the chat box and press Enter — try something like "Explain quantum computing in simple terms"
Watch the response stream in token by token, just like a cloud AI
Ask a follow-up question — the model remembers the conversation context
Click New Chat in the sidebar to start a fresh conversation

Response speed depends on your hardware:

Hardware	Approx. Speed	Notes
RTX 4090	~80 tok/s	Near-instant responses
RTX 3060	~30 tok/s	Smooth conversational speed
CPU only	~5 tok/s	Usable but noticeably slower

Test it: Ask the model "What is 2+2?" — you should get a response within a few seconds on any hardware.

Chat history and models persist locally across sessions. No account or cloud connection needed — everything is stored on your machine.

How Fast Is Fast?

In practical terms, an RTX 4090 generates a full paragraph (~100 tokens) in about 1.5 seconds. An RTX 3060 takes about 4 seconds for the same response. On CPU only, expect around 20 seconds. These times are for the default 3B model — larger models (7B, 13B) are proportionally slower but more capable.

Step 5: Customize

Open WebUI has a number of features that let you tailor the chatbot to your needs.

System Prompts

Set a system prompt to give the model a persistent personality or role. For example: "You are a helpful coding assistant who explains concepts with examples." Click the settings icon in the chat header to edit.

Temperature

Temperature controls randomness. Lower values (0.1–0.3) give focused, deterministic answers. Higher values (0.7–1.0) produce more creative and varied responses. Adjust in the model parameters panel.

Switch Models

Use the model dropdown at the top of the chat to switch between installed models. Larger models (7B, 13B) are more capable but slower; smaller models (1B, 3B) are faster and use less memory.

Upload Documents

Drag and drop a PDF, text file, or code file into the chat to ask questions about it. Open WebUI chunks the document and feeds relevant sections to the model as context (RAG).

Sample System Prompts

# Coding assistant
You are a senior software engineer. Explain concepts with
runnable code examples. Use Rust when possible. Be concise.
 
# Creative writing coach
You are a creative writing mentor. Give constructive feedback
on prose, suggest improvements, and explain writing techniques
with examples from published fiction.

Troubleshooting

Problem	Cause	Fix
Connection refused	Port 11434 already in use	Run `lsof -i :11434` to find the process, then `systemctl stop ollama` (if system-installed) or `kill <PID>`
Model not responding	Model still loading into memory	Wait 10–30 seconds; check the console tab for progress
Very slow responses	Model too large for VRAM; running on CPU	Switch to a smaller model (1B or 3B) or close other GPU apps

Key Takeaways

Everything runs locally — your conversations never leave your machine
Ollama manages model downloads and serving automatically
GPU accelerates responses but is not required
System prompts let you customize the model’s behavior per-chat
You can switch models at any time without restarting

Next Steps

Try image generation — follow the First Image tutorial
Browse more models — see model management for installing larger or specialized models
Create your own model pack — follow the Model Pack tutorial
Try a different LLM engine — see LLM Engines for llama.cpp and cloud API alternatives
Try the AI agent — install apps, debug issues, and generate code with natural language
Read the full documentation for advanced features like API keys and multi-device sync

Learn More

First Image Tutorial

Generate AI images with ComfyUI

First Music Tutorial

Generate music with ACE-Step

User Guide

Full app management documentation

Troubleshooting

GPU issues and benchmarks