Set up your NVIDIA GPU for local AI with the right CUDA version, PyTorch build, and VRAM configuration.

Check Your NVIDIA GPU

Open a terminal and run nvidia-smi. You should see your GPU model, driver version, VRAM amount, and CUDA version. If the command is not found, you need to install the NVIDIA driver first.

Thumper-Run's GPU detection reads this information automatically and displays it in Settings > Hardware.

Supported GPUs

For local AI, you need a GPU with Compute Capability 7.0+ (Turing architecture or newer):

RTX 20-series (Turing): 2060, 2070, 2080 – 6–11 GB VRAM
RTX 30-series (Ampere): 3060, 3070, 3080, 3090 – 8–24 GB VRAM
RTX 40-series (Ada Lovelace): 4060, 4070, 4080, 4090 – 8–24 GB VRAM
RTX 50-series (Blackwell): 5070, 5080, 5090 – 12–32 GB VRAM

Older GTX cards (1060, 1080) work for some tasks but lack tensor cores and have limited VRAM.

Driver Installation

Linux (Ubuntu/Fedora):

`bash

Ubuntu

sudo apt install nvidia-driver-560

sudo reboot

Verify

nvidia-smi

Windows: Download the latest Game Ready or Studio driver from nvidia.com. Studio drivers are more stable for AI workloads.

macOS: NVIDIA GPUs are not supported on modern macOS. Use Apple Silicon (M1+) with Metal acceleration instead.

CUDA Toolkit

Most AI frameworks ship with their own CUDA runtime, so you typically do not need to install the CUDA Toolkit separately. PyTorch bundles CUDA 12.x in its pip wheels. Thumper-Run handles this automatically.

If you need the toolkit for custom development:

`bash

sudo apt install nvidia-cuda-toolkit

nvcc --version # Should show 12.x

PyTorch and CUDA Compatibility

The most common issue is mismatched PyTorch and CUDA versions. Key rules:

PyTorch 2.5+ supports CUDA 12.4
Always use the pip wheel with the right CUDA version: pip install torch --index-url https://download.pytorch.org/whl/cu124
Do not mix conda and pip for PyTorch—this causes DLL conflicts

Thumper-Run's install pipeline selects the correct PyTorch wheel automatically.

VRAM Optimization

Maximize your available VRAM:

Close your browser while running AI—Chrome alone can use 500 MB+ of VRAM
Reduce display resolution if you are VRAM-constrained
Use quantized models (Q4_K_M for LLMs, FP16 for diffusion)
Enable xformers or torch.compile for memory-efficient attention
See the VRAM guide for per-model requirements

Tensor Cores

RTX cards have tensor cores that accelerate AI operations by 2–4x compared to standard CUDA cores. They activate automatically when using FP16 or BF16 precision. If you are running in FP32, you are leaving performance on the table.

Common Issues

"CUDA out of memory" – The model does not fit in your VRAM. Use a smaller model or increase quantization. See our VRAM guide.
"No CUDA GPUs are available" – PyTorch was installed without CUDA support. Reinstall with the correct index URL.
Slow generation despite having a GPU – The model may be falling back to CPU. Check nvidia-smi during inference to verify GPU utilization.
Driver mismatch – Run nvidia-smi to check your driver's maximum CUDA version. PyTorch's CUDA version must be ≤ this number.

For more help, see the troubleshooting guide and the AMD alternative.

Ready to try it? Download Thumper-Run free →