Fine-Tuning LLMs

Four tools dominate the open-source LLM fine-tuning landscape. Each has different strengths depending on your hardware, experience level, and requirements.

Unsloth

Unsloth is the best choice when you have a single GPU and want maximum speed. It uses custom CUDA kernels that deliver 2–5x speedups over standard HuggingFace training, with 40–60% VRAM reduction through intelligent gradient checkpointing and memory management. Recent versions also support vision and audio fine-tuning for multimodal models.

Unsloth uses a dual license: free for personal and research use, commercial license required for production deployments with more than $5M annual revenue. The free tier has no feature restrictions — you get the same speed and VRAM savings either way.

Axolotl

Axolotl is the power-user’s tool. A single YAML configuration file controls every aspect of training: model, dataset, hyperparameters, quantization, distributed strategy, and logging. It supports over 100 model architectures, FSDP and DeepSpeed for multi-GPU training, and integrates with W&B and MLflow for experiment tracking.

Axolotl excels at reproducibility. You can version-control your YAML configs, share them with teammates, and re-run training with identical results. It also supports advanced methods like DPO and ORPO for alignment training.

# Example axolotl config (axolotl.yaml)
base_model: meta-llama/Llama-3.2-8B-Instruct
model_type: LlamaForCausalLM
load_in_4bit: true
adapter: qlora
lora_r: 32
lora_alpha: 64
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj
 
datasets:
  - path: data/training.jsonl
    type: sharegpt
 
sequence_len: 4096
micro_batch_size: 2
gradient_accumulation_steps: 4
num_epochs: 3
learning_rate: 2e-4
lr_scheduler: cosine
warmup_ratio: 0.1
optimizer: adamw_torch
bf16: true
 
wandb_project: my-llm-finetune
output_dir: ./output/llama-3.2-8b-qlora

LLaMA-Factory

LLaMA-Factory is the most accessible option. Its web UI lets you configure training visually, monitor progress in real time, and preview outputs without writing code. The CLI is equally capable for scripted workflows. Choose LLaMA-Factory when you want a GUI-first experience or need to onboard team members who are not comfortable with YAML configs.

H2O LLM Studio

H2O LLM Studio is a no-code web application for fine-tuning. It provides a dashboard for managing experiments, comparing runs, and exporting models. It is the best choice for teams that want a managed UI without the flexibility trade-offs of a config-driven tool.

The format of your training data determines how the model learns. Most fine-tuning tools support several standard formats. Choose the one that best matches your use case.

ShareGPT (Multi-Turn Conversations)

ShareGPT format is ideal for conversational fine-tuning. Each example is a multi-turn dialogue with clear role labels. This is the recommended format for chatbots, assistants, and any multi-turn interaction.

{
  "conversations": [
    {"from": "system", "value": "You are a helpful coding assistant."},
    {"from": "human", "value": "Write a Python function to reverse a string."},
    {"from": "gpt", "value": "def reverse_string(s):\n    return s[::-1]"},
    {"from": "human", "value": "Now make it handle unicode properly."},
    {"from": "gpt", "value": "import unicodedata\n\ndef reverse_string(s):\n    # Normalize to NFC, reverse grapheme clusters\n    normalized = unicodedata.normalize('NFC', s)\n    return ''.join(reversed(list(normalized)))"}
  ]
}

Alpaca (Instruction-Response)

Alpaca format is the simplest structure for instruction-following tasks. Each example has an instruction, optional input context, and expected output.

{
  "instruction": "Summarize the following paragraph in one sentence.",
  "input": "The transformer architecture was introduced in 2017 by Vaswani et al. in their paper 'Attention Is All You Need'. It replaced recurrent layers with self-attention mechanisms, enabling parallel training and better long-range dependency modeling.",
  "output": "The transformer architecture replaced recurrence with self-attention for parallel training and improved long-range modeling."
}

OpenAI-Compatible (Messages)

OpenAI message format works with tools that support the Chat Completions API schema. It is interchangeable with ShareGPT for most training tools.

{
  "messages": [
    {"role": "system", "content": "You are a customer support agent for Acme Corp."},
    {"role": "user", "content": "I can't log into my account."},
    {"role": "assistant", "content": "I can help with that. Could you tell me the email address associated with your account?"}
  ]
}

Raw Text Continuation

Raw text format is used for continued pre-training (domain adaptation) rather than instruction tuning. Each example is a block of unstructured text that the model learns to predict next-token on.

{"text": "The CRDT (Conflict-free Replicated Data Type) is a data structure that can be replicated across multiple nodes, where each node can update its own replica independently and concurrently without coordination. When replicas are merged, all conflicts are resolved automatically and deterministically."}

Multi-Turn Best Practices

• Keep conversations to 3–8 turns — longer conversations dilute the training signal per example
• Include system prompts in every example so the model learns to follow them consistently
• Vary the complexity across turns — mix simple questions with follow-ups that require reasoning
• Remove examples where the assistant gives incorrect or low-quality responses
• Balance your dataset: no single topic should dominate more than 20–30% of examples

Your base model determines the ceiling of your fine-tune. A larger model with better pre-training will produce better results, but requires more VRAM and longer training times. The table below shows approximate VRAM requirements for QLoRA and LoRA fine-tuning.

The right training parameters depend on your dataset size, hardware, and quality targets. This section covers the parameters that have the biggest impact on results.

Quantization (QLoRA)

QLoRA loads the base model in quantized precision and trains LoRA adapters in full precision on top. The quantization format affects both memory usage and training quality.

Effective Batch Size

The effective batch size is the product of your micro batch size and gradient accumulation steps. Larger effective batches smooth the loss curve but slow convergence. A good starting point is 8–32.

# Effective batch = micro_batch * gradient_accumulation
# Example: 2 * 4 = 8 effective batch size
micro_batch_size: 2
gradient_accumulation_steps: 4
 
# If you have more VRAM, increase micro_batch first:
micro_batch_size: 4
gradient_accumulation_steps: 2
# Same effective batch (8) but faster per step

Sequence Length

Sequence length determines the maximum context window during training. Longer sequences use quadratically more VRAM (due to attention). Set this to the longest example in your dataset, rounded up to a power of 2. For most fine-tunes, 2048 or 4096 is sufficient.

LoRA Target Modules

Which layers you apply LoRA to directly affects quality and VRAM usage. The following guidance is based on empirical results from the RunPod and Unsloth communities:

• Minimal (q_proj + v_proj) — lowest VRAM, fastest training, good enough for simple style adaptation
• Standard (q_proj + k_proj + v_proj + o_proj) — balanced choice for most fine-tunes
• Full attention + MLP (all 7 modules) — highest quality, recommended for knowledge injection and complex tasks. Adds gate_proj, up_proj, down_proj

Higher LoRA rank (lora_r) increases adapter expressiveness at the cost of more VRAM. Start with r=16 for experimentation, r=32 for production, r=64+ for complex domain adaptation. Set lora_alpha to 2x your lora_r value.

Learning Rate and Schedule

For QLoRA fine-tuning, start with a learning rate of 2e-4. Use a cosine scheduler with 5–10% warmup ratio. If training is unstable (loss spikes), halve the learning rate. If training is too slow (loss plateaus early), double it.

Evaluation is the most underrated step in fine-tuning. A model that looks good on the loss chart can still produce garbage in production. Use both quantitative metrics and qualitative testing.

Quantitative Metrics

Monitor these metrics during and after training:

• Training loss — should decrease smoothly. Spikes indicate learning rate issues or bad data
• Validation loss — should track training loss. When it diverges (goes up while training loss continues down), you are overfitting. Stop training
• Perplexity — measure before and after fine-tuning on a held-out test set. Lower is better. A good fine-tune reduces perplexity on in-domain text without significantly increasing it on general text

Qualitative Testing

Create a benchmark of 10–20 questions that cover your use case. Score each response on a 1–5 scale. Run the benchmark on the base model, then on your fine-tune, and compare.

# Evaluation template
benchmark:
  - prompt: "Explain CRDT merge semantics in one paragraph."
    criteria: accuracy, conciseness, correct terminology
    base_score: 3
    finetune_score: 5
 
  - prompt: "Write a Rust function to parse a GGUF header."
    criteria: compiles, correct byte offsets, error handling
    base_score: 2
    finetune_score: 4
 
  - prompt: "What is the difference between LoRA and QLoRA?"
    criteria: technically accurate, mentions NF4, VRAM savings
    base_score: 4
    finetune_score: 5

Red Flags

Stop training and investigate if you observe any of these:

• Verbatim repetition — the model outputs exact sentences from training data. This is memorization, not learning. Reduce epochs or increase dataset size
• Catastrophic forgetting — the model loses general capabilities (cannot answer basic questions it could before). Reduce learning rate, reduce epochs, or add general-domain data to your training set
• Wrong format — the model ignores the chat template or outputs in a different format than expected. Check your dataset format matches the base model’s expected template
• Repetition loops — the model gets stuck repeating the same phrase. Usually caused by overfitting on short or repetitive training examples
• Refusal regression — the model refuses legitimate requests it previously handled. Your dataset may contain too many refusal examples

Once training is complete, you need to export your fine-tune into a format that inference engines can load. The typical pipeline is: merge adapter into base model, then convert to GGUF for local deployment.

Merge Adapter

LoRA and QLoRA produce adapter files (typically 50–500 MB) that are applied on top of the base model at load time. For deployment, you can either ship the adapter separately or merge it into the base weights to create a standalone model.

# Merge LoRA adapter into base model
python -m peft.merge_and_unload \
  --base_model meta-llama/Llama-3.2-8B-Instruct \
  --adapter_path ./output/llama-3.2-8b-qlora \
  --output_dir ./output/llama-3.2-8b-merged

GGUF Conversion

Convert the merged model to GGUF format for llama.cpp, Ollama, Jan, and Thumper’s embedded inference. Choose a quantization level based on your deployment target.

# Convert to GGUF with different quantization levels
python llama.cpp/convert_hf_to_gguf.py \
  ./output/llama-3.2-8b-merged \
  --outfile llama-3.2-8b-finetune-f16.gguf \
  --outtype f16
 
# Quantize to Q4_K_M (recommended for most deployments)
llama.cpp/llama-quantize \
  llama-3.2-8b-finetune-f16.gguf \
  llama-3.2-8b-finetune-Q4_K_M.gguf Q4_K_M
 
# Q5_K_M for higher quality (slightly larger)
llama.cpp/llama-quantize \
  llama-3.2-8b-finetune-f16.gguf \
  llama-3.2-8b-finetune-Q5_K_M.gguf Q5_K_M
 
# Q8_0 for maximum quality (largest file)
llama.cpp/llama-quantize \
  llama-3.2-8b-finetune-f16.gguf \
  llama-3.2-8b-finetune-Q8_0.gguf Q8_0

Ollama Modelfile

Create an Ollama Modelfile to register your fine-tune as a named model. This lets you use it with any Ollama-compatible app.

# Create Modelfile
cat > Modelfile << 'EOF'
FROM ./llama-3.2-8b-finetune-Q4_K_M.gguf
 
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 4096
 
SYSTEM "You are a helpful assistant specialized in CRDT systems."
EOF
 
# Register with Ollama
ollama create my-finetune -f Modelfile
 
# Test it
ollama run my-finetune "Explain eventual consistency."

Deployment Options

Once you have a GGUF file, you can deploy it through several channels:

• Ollama — create a Modelfile and register with ollama create. Works with Open WebUI, SillyTavern, and any OpenAI-compatible client
• Jan — copy the GGUF file to Jan’s model directory. It appears automatically in the model selector
• Thumper Marketplace — package as a model manifest and sell through the Thumper catalog. See the Selling Fine-Tunes guide
• API serving — use llama-server (llama.cpp’s built-in server) or vLLM for production API endpoints