Fine-Tuning Image Models

There are several approaches to customizing image generation models. Each makes different trade-offs between output quality, training cost, composability, and hardware requirements.

Decision Flowchart

Start with LoRA — it covers 90% of use cases with the best balance of quality, speed, and composability. Only move to other methods when LoRA falls short:

• Need to stack multiple concepts? → LoRA (composable, apply multiple at inference)
• Only 6–8 GB VRAM? → QLoRA (quantized base model during training)
• Need exact likeness of a face/object? → DreamBooth + LoRA (DreamBooth precision, LoRA file size)
• Just need a trigger word for a concept? → Textual Inversion (tiny output, fast to share)
• LoRA can’t capture the style complexity? → LyCORIS (LoHA or LoKr for higher expressiveness)
• Entire model needs to change domain? → Full Fine-Tune (requires 24+ GB VRAM or cloud)

Dataset quality is the single biggest factor in training results. A well-curated 20-image dataset beats a sloppy 200-image one every time.

Resolution by Model

Captioning Strategies

Every training image needs a text caption describing its content. The captioning strategy you choose affects how the model learns and how you prompt it later.

Auto-Captioning Tools

Manual captioning is most accurate but time-consuming. These tools auto-generate captions that you can review and refine:

Multi-Concept Directory Structure

When training multiple concepts (e.g., a character and a style), organize images into separate directories with repeats:

# Directory structure for multi-concept training
dataset/
  10_sks_character/        # 10 repeats, trigger: sks_character
    image01.png
    image01.txt            # caption file
    image02.png
    image02.txt
    ...
  5_xyz_style/             # 5 repeats, trigger: xyz_style
    style01.png
    style01.txt
    ...
  reg/                     # regularization images (optional)
    1_person/
      reg001.png
      ...

The number prefix (e.g., "10_") controls how many times each image is repeated per epoch. Use higher repeats for smaller concept sets and lower repeats for larger ones to balance training.

Regularization Images

Regularization images prevent "language drift" — where the model forgets what a generic person/object looks like and always generates your trained subject. Generate 200–500 images using the base model with generic prompts matching your subject class (e.g., "a photo of a person" for a character LoRA).

Trigger Words

A trigger word is a unique token that activates your trained concept at inference time. Choose something that doesn’t exist in the model’s vocabulary to avoid collisions:

Place the trigger word at the beginning of every caption. Consistency is critical: if some captions omit the trigger word, the model may not learn to associate it with your concept.

Image Preprocessing

Before adding images to your dataset, preprocess them for consistency:

• Remove images with text overlays, watermarks, or borders
• Crop to focus on the subject (leave some context but remove unnecessary background)
• Ensure consistent lighting direction across the set if possible
• Remove near-duplicate images — they waste training compute on redundant data
• For face training: include at least 3 different angles (front, 3/4 view, profile)

Dataset Quality Scoring

Before training, score your dataset against this checklist. Aim for all items checked:

• All images at or above native resolution (no upscaled low-res)
• Consistent subject across all images (same character/style/concept)
• Variety of angles, lighting, and backgrounds
• No watermarks, text overlays, or compression artifacts
• Every image has a reviewed caption (auto-generated captions manually checked)
• Trigger word consistently present in all captions
• 15–50 images for characters, 50–200 for styles
• Regularization images prepared (for subject LoRAs)

Kohya SS is the most widely used LoRA training tool. It provides a GUI that maps to the underlying training script parameters. Here’s a tab-by-tab walkthrough of the key settings.

Source Model Tab

• Pretrained model — path to the base model checkpoint (e.g., sd_xl_base_1.0.safetensors)
• Model type — SD 1.5, SD 2.x, or SDXL (must match your checkpoint)
• V2 / V-parameterization — enable only for SD 2.x models
• Save precision — fp16 recommended (half the file size, negligible quality loss)

Folders Tab

• Image folder — parent directory containing your concept subfolders (e.g., dataset/)
• Regularization folder — directory with class-generic images (leave empty to skip)
• Output folder — where trained LoRA files are saved
• Log folder — TensorBoard logs for monitoring training

Parameters Tab

• Learning rate — controls how fast the model learns (see Training Parameters section)
• Network rank (dim) — LoRA rank; higher = more expressive but larger file
• Network alpha — scaling factor; typically set equal to rank or half of rank
• Batch size — images per step; limited by VRAM
• Epochs — number of full passes through the dataset
• Optimizer — algorithm for weight updates (see Optimizer Selection section)
• LR scheduler — how learning rate changes over time (cosine, constant, linear)

Advanced Tab

• Noise offset — improves dark/light scene generation (see Optimizer Selection)
• Clip skip — skip final CLIP layers; 1 for realism, 2 for anime
• Mixed precision — fp16 or bf16 to reduce VRAM usage
• Gradient checkpointing — trades speed for VRAM; enable on 8–12 GB cards
• Cache latents — pre-compute VAE latents; saves VRAM and speeds up training
• Shuffle captions — randomize tag order per step (recommended for booru-style captions)

Complete Field Reference

Here is a quick-reference table of the most commonly adjusted fields across all tabs:

These recommended parameters are starting points organized by VRAM tier. Adjust based on your dataset size and target quality.

The total number of training steps is: (images × repeats × epochs) / batch_size. For a 30-image dataset with 10 repeats, 10 epochs, and batch size 1, that’s 3,000 steps — typically a good starting point.

The optimizer controls how weight updates are computed. The right choice depends on whether you want to tune the learning rate manually or let the optimizer handle it.

Noise Offset by Style

Noise offset helps the model generate very dark or very bright scenes that are underrepresented in typical training data. The ideal value depends on your target art style:

Trained LoRAs are saved in SafeTensors format (.safetensors). This is the standard format supported by all major inference tools.

Using LoRAs in Different Tools

Strength Guidance

LoRA strength (also called weight) controls how strongly the LoRA affects the output. The ideal range depends on how the LoRA was trained:

• 0.5–0.7 — subtle influence; good for style mixing
• 0.7–1.0 — standard range for most LoRAs
• 1.0–1.5 — strong effect; may cause artifacts if LoRA was overtrained

Stacking Multiple LoRAs

LoRAs are composable — you can apply multiple simultaneously. Rules for stacking:

• Keep total combined weight under 1.5 (e.g., two LoRAs at 0.7 each = 1.4 total)
• Style + subject LoRAs stack well; two subject LoRAs often conflict
• Apply the most important LoRA first (highest in the chain/prompt)
• If results degrade, reduce individual weights before removing a LoRA

Base Model Compatibility

LoRAs are locked to their base model architecture. An SDXL LoRA only works with SDXL-based models; an SD 1.5 LoRA only works with SD 1.5-based models. Custom finetunes of the same architecture (e.g., DreamShaper XL, which is based on SDXL) are compatible.

Training a LoRA is only half the work. Systematic evaluation tells you whether the training was successful and whether more epochs or parameter changes would help.

Visual A/B Testing

Generate the same prompt with and without the LoRA, using the same seed. Compare side by side:

• Does the LoRA add the intended concept without degrading other elements?
• Is the trigger word necessary to activate the concept, or does it leak into all generations?
• Are colors, composition, and detail level consistent with the base model?

Generalization Test

Test the LoRA with prompts it was NOT trained on. A well-trained LoRA generalizes to new contexts:

# If you trained a character LoRA, try:
sks_character in a spacesuit, standing on Mars, dramatic lighting
sks_character as a medieval knight, oil painting style
sks_character reading a book in a cozy library, warm lighting
 
# If you trained a style LoRA, try:
a cyberpunk city at night, neon lights, xyz_style
a still life of fruit on a table, xyz_style
a portrait of an elderly man, xyz_style

If the LoRA only works with prompts very similar to the training data, it may be overtrained. Reduce epochs or increase dataset diversity.

Strength Sweep

Generate the same prompt at strengths 0.4, 0.6, 0.8, 1.0, and 1.2 using the same seed. A healthy LoRA shows a smooth gradient from subtle to strong effect. Red flags:

• Artifacts or color shifts at 0.8+ → overtrained
• No visible effect until 1.0+ → undertrained or rank too low
• Sudden quality collapse at 1.2 → normal; stay within 0.6–1.0 range

Checkpoint Comparison

If you saved checkpoints at multiple epochs (recommended: every 2–3 epochs), compare them with the same prompt and seed. Pick the epoch that gives the best balance between concept fidelity and generalization. Earlier checkpoints generalize better; later ones capture the concept more precisely.

TensorBoard Monitoring

Kohya SS outputs TensorBoard logs that let you monitor training in real-time. Key metrics to watch:

# Launch TensorBoard to view training logs
tensorboard --logdir /path/to/log/folder --port 6006
# Then open http://localhost:6006 in your browser

Stop training early if loss starts increasing (overfitting) or if visual inspection of checkpoint outputs shows degradation. The last checkpoint before quality drops is typically the best.

Once you’re comfortable with basic LoRA training, these techniques can improve results for specific use cases.

Network Types (LyCORIS)

LyCORIS is a family of LoRA-like methods that decompose weight updates differently. Available types in Kohya SS:

Aspect Ratio Bucketing

Instead of cropping all images to squares, aspect ratio bucketing groups images by similar dimensions and trains on each group. This preserves composition and avoids cropping out important content.

• Enable in Kohya SS: check "Enable bucket" in the Parameters tab
• Set min/max resolution to match your model (e.g., 512–1536 for SDXL)
• Bucket resolution step: 64 (default; smaller steps = more buckets = more VRAM)

Caption Dropout

Randomly dropping captions during training forces the model to associate the visual concept with the trigger word alone, improving trigger word reliability. Set caption dropout rate to 0.05–0.1 (5–10% of steps use empty captions).

Tag Weighting

When using booru-style captions, you can weight specific tags to control training emphasis. In Kohya SS, use parentheses: "(important_tag:1.5), normal_tag, (minor_tag:0.5)". Higher weights make the model pay more attention to that tag during training.

Target Modules Selection

By default, LoRA trains on the UNet attention layers. You can target additional modules for different effects:

• UNet attention only (default) — fastest training, good for most styles and subjects
• UNet attention + text encoder — better prompt response; use lower LR for text encoder (5e-5)
• All UNet layers — maximum expressiveness; larger file, longer training
• Conv layers — can help with texture/pattern-heavy concepts

FLUX-Specific Training

FLUX models use a different architecture (DiT/MMDiT) than Stable Diffusion (UNet). Key differences for LoRA training:

• FLUX requires more VRAM: 12 GB minimum for QLoRA, 24 GB for standard LoRA
• Use lower learning rates (1e-5 to 5e-5) compared to SDXL (1e-4)
• Training is slower per step but often needs fewer total steps
• Captions should be natural language (FLUX was trained on detailed descriptions, not tags)
• Rank 16–32 is usually sufficient; FLUX responds well to low-rank adaptation

Multi-Resolution Training

For maximum quality, train at multiple resolutions simultaneously. This teaches the model to generate consistent results across different output sizes:

• Enable aspect ratio bucketing (covers resolution variety)
• Provide images at the highest resolution you can (the trainer downsamples as needed)
• For SDXL: include images in portrait (768×1344), landscape (1344×768), and square (1024×1024) orientations