GramSR

gramsr.md

GramSR, DINOv3, and Texture‑Level Conditioning — A Technical Summary

(from Bing Copilot)

1. Background: Why GramSR Exists

Diffusion‑based super‑resolution (SR) models often struggle to balance:

• pixel‑level fidelity (structure, geometry)
• semantic detail (perceptual sharpness)
• texture realism (material appearance, micro‑patterns)

Prior “dual‑LoRA” frameworks (e.g., PiSA‑SR) addressed the first two by training:

1. Pixel‑LoRA (MSE‑driven)
2. Semantic‑LoRA (LPIPS/CSD‑driven)

But they lacked a mechanism for texture statistics.
GramSR introduces a third LoRA trained using Gram‑matrix losses computed from DINOv3 features.

---

2. What Dual‑LoRA Actually Means

In PiSA‑SR and similar methods:

• There are two physically separate LoRA modules, not one LoRA with two losses.
• Each LoRA is trained independently with its own objective.
• At inference, the deltas are combined with separate guidance scales.

GramSR extends this to three LoRAs (pixel, semantic, texture).

---

3. Why Gram‑Matrix Loss?

Gram matrices capture second‑order feature correlations, originally introduced in:

• Gatys et al. (2016) — Neural Style Transfer
  https://arxiv.org/abs/1508.06576

GramSR cites two sources for Gram‑based texture modeling:

1. Gatys et al. (foundational)
2. DINOv3 (modern use of Gram correlations in ViT features)
   https://ai.meta.com/research/dinov3

DINOv3 does not introduce Gram loss, but uses Gram‑like feature correlation analysis, making it a natural encoder for texture statistics.

---

4. Why DINOv3 Specifically?

DINOv3 is chosen because its ViT features have properties ideal for Gram‑matrix texture modeling:

• Global receptive field from the first layer
• Strong long‑range correlations
• Homogeneous patch embeddings
• Clean covariance structure
• Excellent separation of structure vs texture

These properties make Gram matrices computed from DINOv3 stable, expressive, and texture‑sensitive.

Could other SSL encoders work?

Yes, but with worse texture statistics:

• Good but weaker: DINOv2, MAE, iBOT, MoCo‑v3 (ViT)
• Poor: CNN‑based SSL (MoCo‑v1/v2, BYOL, SimCLR)

CNNs produce local, spatially biased, noisier Gram matrices.

---

5. Why Not Use DINOv3 for All Perceptual Losses?

GramSR uses:

• LPIPS → VGG
• CSD → classifier
• Gram loss → DINOv3

Reason:
DINOv3 features are too semantic and too global for LPIPS‑style perceptual similarity.
They are not sensitive to:

• edges
• local distortions
• micro‑structure
• pixel‑aligned differences

VGG‑based LPIPS remains superior for local perceptual fidelity.

---

6. CNN‑Only Inference: How to Adapt GramSR

If your hardware cannot run transformers at inference, you can still use GramSR’s ideas:

Training (heavy, offline):

• Use DINOv3 only as a frozen loss network:
  • Gram‑matrix texture loss
  • Optional DINO‑based perceptual loss (mid‑level only)
• Use VGG LPIPS for local perceptual structure
• Train a pure CNN SR model (e.g., BasicVSR++, EDVR, or custom CNN)

Inference (light, hardware‑friendly):

• Only the CNN runs
• No DINOv3
• No VGG
• No transformers at all

You keep the benefits of DINO‑based texture modeling without paying inference cost.

---

7. Can We Train a DINO‑Like Model That Focuses on Local Texture?

Yes — but you must modify the training objective, not the architecture.

Effective strategies:

1. Small‑crop training

   • Use 32–64 px crops
   • Reduce invariance
   • Forces sensitivity to micro‑texture

2. Local contrastive objectives

   • Patch‑to‑patch matching
   • Region‑level alignment
   • Similar to DenseCL, PixPro, iBOT local loss

3. Texture‑classification auxiliary tasks

   • Material categories (metal, wood, fabric, plastic)
   • Can be self‑supervised via clustering

4. Gram‑matrix consistency loss

   • Encourage stable second‑order correlations
   • Works well with CNNs

Result:

You get a CNN‑based texture encoder that approximates DINOv3’s texture sensitivity but is hardware‑friendly.

---

8. Why There Is No “CNN Version of DINOv3”

DINOv3 relies on:

• global self‑attention
• patch tokenization
• masked modeling
• global feature alignment

A CNN cannot replicate these without effectively becoming a transformer.
Thus, a “CNN DINOv3” does not exist — but a CNN trained with DINO‑style local objectives can achieve similar texture sensitivity.

---

9. Practical Takeaways

• GramSR’s novelty is the texture‑level LoRA trained with DINOv3 Gram loss.
• DINOv3 is used only where it excels: texture statistics, not perceptual similarity.
• You can build a CNN‑only SR model that benefits from DINOv3 during training.
• You can train a CNN‑based texture encoder using local SSL objectives if DINOv3 is too heavy.
• Gram‑matrix losses remain the most effective way to encode material appearance.

---

External Links

• GramSR paper:
  https://arxiv.org/abs/2604.25457

• DINOv3:
  https://ai.meta.com/research/dinov3

• LPIPS (VGG perceptual loss):
  https://github.com/richzhang/PerceptualSimilarity

• BasicVSR++ (CNN SR backbone):
  https://arxiv.org/abs/2104.13371