GPU Selection
Your GPU determines what you can train. VRAM is the primary bottleneck — it must hold the model weights, optimizer states, gradients, and activations all at once. Use the checker below to see what your current hardware supports.
| GPU | Status | Backend | Image Gen | LLM Speed | Notes |
|---|---|---|---|---|---|
| RTX 4090 (24 GB) | Full | CUDA 12.4 | ~3s | ~120 tok/s | Fastest consumer GPU |
| RTX 4070 (12 GB) | Full | CUDA 12.4 | ~8s | ~80 tok/s | Great balance of price/performance |
| RTX 3060 (12 GB) | Full | CUDA 11.8 | ~15s | ~45 tok/s | 12 GB VRAM at budget price |
| GTX 1660 (6 GB) | Partial | CUDA 11.8 | ~30s | ~20 tok/s | 6 GB limits model size |
| RX 7900 XT (20 GB) | Full | ROCm 6.2 | ~6s | ~90 tok/s | Best AMD option, large VRAM |
| RX 7600 (8 GB) | Full | ROCm 6.2 | ~18s | ~40 tok/s | Budget AMD with ROCm support |
| Radeon 780M APU (8 GB shared) | Partial | ROCm 6.2 | ~45s | ~15 tok/s | BF16 only, 5-min MIOpen warmup |
| Arc A770 (16 GB) | Partial | oneAPI/IPEX | ~20s | ~35 tok/s | Requires oneAPI runtime |
| M2 Pro (16 GB unified) | Full | MPS (Metal) | ~12s | ~50 tok/s | Unified memory, no discrete VRAM limit |
| M1 (8 GB unified) | Partial | MPS (Metal) | ~35s | ~25 tok/s | 8 GB tight for SDXL |
| M3 Max (36 GB unified) | Full | MPS (Metal) | ~8s | ~70 tok/s | Runs large models easily |
| CPU only (no GPU) | Partial | CPU fallback | ~180s | ~5 tok/s | Works but 10-50x slower |
The tables below show minimum VRAM requirements for each training method. If your GPU doesn’t meet the minimum, the training job will fail with an out-of-memory error.
Image Training VRAM
Image model training requirements vary significantly by method. LoRA is the most VRAM-efficient, while full fine-tuning requires high-end hardware.
| Method | 8 GB | 12 GB | 16 GB | 24 GB |
|---|---|---|---|---|
| LoRA SD 1.5 | Yes | Yes | Yes | Yes |
| LoRA SDXL | No | Yes | Yes | Yes |
| DreamBooth SD 1.5 | No | Yes | Yes | Yes |
| DreamBooth SDXL | No | No | Yes | Yes |
| Full Fine-Tune SD 1.5 | No | No | Yes | Yes |
| Textual Inversion | Yes | Yes | Yes | Yes |
LLM Training VRAM
Language model training is more VRAM-hungry than image training due to larger model sizes and longer sequence lengths. QLoRA’s 4-bit quantization is the key enabler for consumer GPUs.
| Method | 8 GB | 12 GB | 16 GB | 24 GB | 48 GB |
|---|---|---|---|---|---|
| QLoRA 3B | Yes | Yes | Yes | Yes | Yes |
| QLoRA 7–8B | No | Yes | Yes | Yes | Yes |
| QLoRA 13B | No | No | Yes | Yes | Yes |
| LoRA 3B | No | Yes | Yes | Yes | Yes |
| LoRA 7–8B | No | No | Yes | Yes | Yes |
| LoRA 13B | No | No | No | Yes | Yes |
| Full 7B | No | No | No | No | Yes |
Multi-GPU Training
Most consumer users have a single GPU, but if you have two or more, some tools can split training across them for larger models or faster throughput.
Supported Tools
- Axolotl — Supports FSDP (Fully Sharded Data Parallel) and DeepSpeed ZeRO stages 1–3
- LLaMA-Factory — Supports DeepSpeed ZeRO stages 1–3 via configuration
When Multi-GPU Matters
- Training models larger than your single GPU’s VRAM (e.g., full 13B fine-tune)
- Reducing training time by splitting batches across GPUs
- Using multiple smaller GPUs you already own (e.g., 2× RTX 3060 12 GB = 24 GB effective)
Cost Comparison
Local training pays for itself quickly. Here’s how local GPU costs compare to popular cloud platforms for two common training jobs.
| Platform | Image LoRA (SDXL, 1 hr) | LLM QLoRA (7B, 4 hr) |
|---|---|---|
| Local RTX 4090 | ~$0.05 (electricity) | ~$0.20 (electricity) |
| RunPod A100 (80 GB) | ~$2.00 | ~$8.00 |
| Replicate | ~$3.50 | ~$14.00 |
After 10–20 training runs, a local GPU has paid for itself compared to cloud pricing. The savings compound with experimentation — you can iterate freely without watching a billing meter.
AMD & Apple Notes
Training tool support varies by GPU vendor. NVIDIA has the broadest compatibility, but AMD and Apple Silicon are catching up.
AMD Discrete GPUs (ROCm)
- ROCm 6.0+ is required for training. Older versions have critical bugs in gradient computation.
- Kohya SS and LLaMA-Factory both support ROCm with the correct PyTorch build.
- Expect 5–15% slower training vs equivalent NVIDIA GPU due to less optimized kernels.
- RX 7900 XT (20 GB) and RX 7900 XTX (24 GB) are the best consumer AMD training GPUs.
AMD APU Warning
Apple Silicon (MPS)
- MPS (Metal Performance Shaders) backend works for inference but training support is partial.
- Kohya SS has experimental MPS support for LoRA training on M1/M2/M3.
- Expect some operations to fall back to CPU, reducing effective throughput.
- Unified memory means no VRAM limit — but bandwidth is shared with the CPU.
Key Takeaways
- VRAM is the primary bottleneck — check the tables before starting a training job
- QLoRA makes 7B LLM training possible on 12 GB GPUs; Unsloth can push it to 8 GB
- Local training costs pennies per run vs dollars per hour in the cloud
- AMD ROCm and Apple MPS require BF16 precision — never use FP16 for training
- APU training is not recommended — use CPU mode instead for integrated graphics