Fine-Tune an LLM

Before you begin, make sure you have:

• GPU — NVIDIA with CUDA 11.8+ and at least 12 GB VRAM
• Training data — 100–5000 conversation examples in JSON format
• LLaMA-Factory — install from the Thumper-Run catalog
• Base model — a HuggingFace model ID (e.g. meta-llama/Llama-2-7b-chat-hf or mistralai/Mistral-7B-Instruct-v0.3)
• Disk space — at least 30 GB free for the base model, adapters, and merged output
• Ollama — for deploying the finished model (install from catalog or ollama.com)

What is QLoRA?

QLoRA (Quantized Low-Rank Adaptation) combines two techniques: 4-bit quantization of the base model and LoRA adapter training. The base model is loaded in NF4 format (using ~4 GB for a 7B model instead of ~14 GB), while the small LoRA adapter matrices are trained in full precision.

This means you get nearly the same training quality as full LoRA at roughly half the VRAM cost. The trade-off is slightly slower training speed due to the quantization/dequantization overhead during forward passes.

Choosing a Base Model

Your base model determines the starting capabilities and chat format of your fine-tuned model:

LLaMA-Factory supports several dataset formats. Choose the one that best matches your data:

ShareGPT Format (Multi-Turn)

Best for training conversational chatbots with multi-turn dialogue:

[
  {
    "conversations": [
      {"from": "human", "value": "What causes rust on steel?"},
      {"from": "gpt", "value": "Rust forms when iron in steel reacts with oxygen and moisture..."},
      {"from": "human", "value": "How do I prevent it?"},
      {"from": "gpt", "value": "Common prevention methods include galvanizing, painting..."}
    ]
  }
]

Alpaca Format (Single Instruction)

Best for instruction-following tasks and Q&A pairs:

[
  {
    "instruction": "Explain the difference between TCP and UDP.",
    "input": "",
    "output": "TCP is a connection-oriented protocol that guarantees delivery..."
  }
]

OpenAI Format (Compatible)

Familiar format if you’re coming from the OpenAI fine-tuning API:

[
  {
    "messages": [
      {"role": "system", "value": "You are a metallurgy expert."},
      {"role": "user", "value": "What causes rust on steel?"},
      {"role": "assistant", "value": "Rust forms when iron reacts..."}
    ]
  }
]

Dataset Quality Tips

• Hold out 10% for evaluation — split your data into train (90%) and test (10%) sets before training
• Consistent formatting — answers should follow a consistent style and tone
• Diverse questions — cover the full range of topics you want the model to handle
• Accurate answers — the model will learn to reproduce your training data, including any errors
• Reasonable length — avoid extremely long responses that exceed the model’s context window

Dataset Size Guidelines

Common Dataset Mistakes

• Inconsistent formatting — mixing markdown, plain text, and HTML in responses confuses the model
• Duplicate entries — exact duplicates cause the model to memorize rather than generalize
• Too-short answers — one-word or one-sentence answers teach terse behavior
• Factual errors — the model will confidently reproduce any errors in the training data
• Missing system prompts — if your use case needs a system prompt, include it in training

LLaMA-Factory provides a web GUI for configuring and running LLM fine-tuning jobs.

1. Open the Thumper-Run catalog and search for LLaMA-Factory
2. Click Install — the pipeline clones the repository, creates a venv, and installs PyTorch + dependencies
3. Wait for installation to complete (typically 5–10 minutes depending on internet speed)
4. Click Launch to start the LLaMA-Factory GUI

The GUI opens in a browser tab. Expected console output:

# Expected console output
LLaMA-Factory starting...
Loading training modules...
CUDA device detected: NVIDIA GeForce RTX 4070 (12 GB)
Running on local URL: http://localhost:7861

GUI Overview

The LLaMA-Factory GUI has tabs for model selection, dataset configuration, training parameters, evaluation, and export. The workflow follows a left-to-right progression through these tabs.

Register Your Dataset

LLaMA-Factory needs to know about your dataset before training. Place your JSON file in the data/ directory and register it in data/dataset_info.json:

{
  "my_domain_data": {
    "file_name": "my_domain_data.json",
    "formatting": "sharegpt",
    "columns": {
      "messages": "conversations"
    }
  }
}

After registration, your dataset appears in the GUI’s dataset dropdown. You can combine multiple datasets in a single training run by selecting them together.

With your dataset ready and LLaMA-Factory running, configure the training job.

QLoRA vs LoRA Parameters

Quantization Options

QLoRA quantizes the base model weights to reduce VRAM usage during training:

• NF4 (Normal Float 4-bit) — the default and recommended option. Optimized distribution for neural network weights.
• FP4 (Float Point 4-bit) — alternative 4-bit format. Slightly different precision characteristics.
• 8-bit — uses more VRAM but preserves more information from the base model. Use if you have VRAM to spare.

Target Modules

LoRA adapters are applied to specific attention layers in the transformer. The target modules control which layers are trained:

• Minimal (q_proj, v_proj) — trains only the query and value projections. Fewest trainable parameters, fastest training, least VRAM.
• Standard (q_proj, k_proj, v_proj, o_proj) — trains all attention projections. Good balance of quality and efficiency.
• Full (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj) — trains attention and MLP layers. Best quality but more VRAM and slower.

Trainable Parameters

The percentage of the model’s parameters that are actually updated during training depends on rank and target modules:

# Approximate trainable parameters for a 7B model
Rank 16, q_proj+v_proj:    ~0.1% (~8M params)
Rank 32, q_proj+v_proj:    ~0.2% (~16M params)
Rank 16, all attention:    ~0.2% (~16M params)
Rank 32, all attn+MLP:     ~0.8% (~56M params)

Training Monitoring

Watch these metrics during training to ensure things are progressing correctly:

• Training loss — should decrease steadily. A good final loss is typically 0.5–1.5 depending on your data.
• Eval loss — should track training loss. If eval loss rises while training loss drops, you’re overfitting.
• Learning rate schedule — visible in the training log. Cosine decay is the default.

Gradient Accumulation

If your GPU cannot fit the desired batch size, use gradient accumulation to simulate larger batches without increasing VRAM usage:

# Effective batch size = batch_size × gradient_accumulation_steps
 
# Example: simulate batch size 16 on a 12 GB GPU
batch_size = 2
gradient_accumulation_steps = 8
# effective_batch = 2 × 8 = 16

Larger effective batch sizes produce smoother gradients and more stable training, at the cost of longer time per optimization step.

Learning Rate Scheduling

LLaMA-Factory supports several learning rate schedulers:

• cosine — gradually decreases LR following a cosine curve. The default and recommended option.
• linear — linearly decreases LR from the initial value to zero.
• constant — keeps LR fixed throughout training. Simple but may not converge as well.
• constant_with_warmup — constant LR after a brief warmup period. Good for short training runs.

A warmup period (typically 3–10% of total steps) gradually increases the learning rate from zero. This prevents early training instability and is enabled by default.

Once training is complete, you need to merge the adapter into the base model, convert to GGUF format, and load it into Ollama for inference.

Merge the Adapter

LLaMA-Factory’s export tab merges the LoRA adapter weights back into the base model, producing a full-size model:

1. Go to the Export tab in LLaMA-Factory
2. Select your trained adapter checkpoint
3. Choose an output directory for the merged model
4. Click Export — this takes 2–5 minutes depending on model size

Convert to GGUF

GGUF is the format used by llama.cpp and Ollama. Convert the merged model with quantization:

# Clone llama.cpp if you don't have it
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
 
# Convert HF model to GGUF
python convert_hf_to_gguf.py /path/to/merged-model \
  --outfile my-model-f16.gguf --outtype f16
 
# Quantize to reduce size (choose one)
./llama-quantize my-model-f16.gguf my-model-Q4_K_M.gguf Q4_K_M

Common quantization levels:

Create an Ollama Modelfile

Create a Modelfile that tells Ollama how to load and configure your model:

# Modelfile
FROM ./my-model-Q4_K_M.gguf
 
TEMPLATE """{{ if .System }}<|system|>
{{ .System }}</s>{{ end }}<|user|>
{{ .Prompt }}</s>
<|assistant|>"""
 
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER stop "</s>"
PARAMETER stop "<|user|>"
 
SYSTEM """You are a helpful domain expert assistant."""

Deploy to Ollama

# Create the model in Ollama
ollama create my-domain-expert -f Modelfile
 
# Test it
ollama run my-domain-expert "What causes rust on steel?"

Evaluation: Base vs Fine-Tuned

Compare your fine-tuned model against the base model with a structured evaluation:

1. Prepare 10 domain-specific questions from your held-out test set
2. Run each question through both the base model and your fine-tuned model
3. Score each answer on accuracy (0–5), relevance (0–5), and fluency (0–5)
4. Calculate the average score for each model
5. A successful fine-tune should show a clear improvement in accuracy and relevance on domain questions

If the fine-tuned model scores lower on general questions, that’s expected — some general capability is traded for domain expertise. This is acceptable as long as the domain improvement is significant.

Chat Template Matching

Each base model family uses a different chat template. Your Modelfile TEMPLATE must match exactly, or the model will produce garbled output. Common templates:

• Llama 3 — uses <|begin_of_text|><|start_header_id|>system<|end_header_id|> format
• Mistral — uses [INST] ... [/INST] format
• ChatML — uses <|im_start|>system ... <|im_end|> format (Qwen, Yi)

LLaMA-Factory handles the chat template automatically during training. The critical step is matching it in your Ollama Modelfile. Check the base model’s HuggingFace page for the exact template.

Alternative Tools

LLaMA-Factory is our recommended starting point, but other tools may suit specific needs:

Recommended Configurations

Ollama Integration with Thumper-Run

Once your model is registered with Ollama, any Thumper-Run app that uses Ollama will automatically detect it:

• SillyTavern — your model appears in the model selector for chat
• Open WebUI — visible in the models dropdown
• Any Ollama-compatible app — uses the standard Ollama API at localhost:11434

You can also use the model via the Ollama API for programmatic access:

# API call to your fine-tuned model
curl http://localhost:11434/api/chat -d '{
  "model": "my-domain-expert",
  "messages": [{"role": "user", "content": "Your question here"}]
}'