Your GPU’s VRAM determines which workloads can run entirely on-device. The table below shows what each tier can handle comfortably. Thumper automatically selects model variants that fit your available VRAM.

ComfyUI’s model_management.py implements a smart memory system that keeps recently-used models in VRAM and only evicts enough to fit the next model. This dramatically speeds up multi-stage pipelines (e.g., text encoder → diffusion → VAE) because models stay warm between stages.

How Smart Memory Works

• Models are loaded into VRAM and tracked by last-use timestamp
• When a new model needs VRAM, the least-recently-used model is partially evicted
• Only enough VRAM is freed to fit the incoming model — the rest stays cached
• On the next generation, warm models skip the load step entirely

--disable-smart-memory

This flag forces full VRAM cleanup between model loads. It prevents OOM crashes when the partial eviction leaves too much residual data, but adds 50–70 seconds of model reload overhead per generation.

AMD APUs (integrated GPUs sharing system RAM) require specific tuning that differs from discrete GPU setups. Thumper applies these settings automatically via platform patches, but understanding them helps with troubleshooting.

BF16 is Mandatory

FP16 causes NaN errors in DiT attention layers because models trained in BF16 overflow FP16’s maximum range (~65,504). Always use BF16 or FP32 on AMD APUs. Thumper’s patches set this automatically.

MIOpen Find-DB Configuration

MIOpen benchmarks GPU kernels to find the fastest algorithm for each operation. The results are cached in a find-database. Configuration is critical for launch time.

Non-Tiled VAE

Non-tiled VAE decoding is up to 1000x faster than tiled decoding for short content (1 second vs 18 minutes). However, non-tiled VAE for long content (180s songs) can require ~40 GB of VRAM. Thumper auto-falls back to tiled VAE for content longer than 60 seconds.

Model loading time is the single biggest contributor to perceived latency. Understanding the loading pipeline helps set expectations and optimize your workflow.

First-Run Penalty

• First launch after install downloads models (minutes to hours depending on connection)
• First inference on AMD triggers MIOpen kernel benchmarking (~2–8 minutes one-time)
• Subsequent launches skip both steps and load from cache

Keeping Models Warm

Once a model is loaded into VRAM, subsequent generations using the same model are nearly instant. Thumper’s smart memory keeps models cached between generations. Tips for staying warm:

• Batch similar workloads together (all image gen, then all LLM, etc.)
• Avoid switching between apps that use different models frequently
• Close unused apps to free VRAM for the active workload

ComfyUI Execution Cache

ComfyUI caches node execution results by input hash. If your inputs have not changed (same prompt, same seed, same model), ComfyUI returns the cached result instantly. Change the seed or prompt to force re-execution.

Model files range from 2 GB to 40+ GB. Disk speed directly impacts load times, especially on first launch when models are read from storage into VRAM.

HuggingFace Cache Deduplication

The HuggingFace cache (~/.cache/huggingface/hub/) uses content-addressable blob storage. Multiple model revisions that share the same files (common for minor updates) use hard links, avoiding duplicate disk usage. Do not copy files out of the cache — Thumper symlinks to them directly.

Sharing HF Cache Across Users

Set HF_HOME to a shared directory (e.g., /shared/huggingface) to share downloaded models across user accounts on the same machine. All Thumper instances will discover models in this shared location automatically.

export HF_HOME=/shared/huggingface