SVDQuant: 4-Bit Quantization Powers 12B Flux on a 16GB 4090 GPU with 3x Speedup

While I think I agree that there are many posts here on HackerNews announcing a new model compression technique, your characterization above understates the technical innovations and practical impacts described in this MIT paper.

Unlike traditional model compression work that simply applies existing techniques, SVDQuant synthesizes several ideas in a comprehensive new approach to model quantization:

- Developing a novel outlier absorption mechanism using low-rank decomposition — this aspect alone seems quite novel, although the math is admittedly way beyond my level

- Combining SVD with smoothing in a way that specifically addresses the unique challenges of diffusion models

- Creating an innovative kernel fusion technique (they call it “Nunchaku”) that makes the theoretical benefits practically realizable, because without this, the extra computation required to implement the above steps would simply slow the model back down to baseline

This isn't just incremental improvement - the paper achieves several breakthrough results:

- First successful 4-bit quantization of both weights AND activations for diffusion models

- 3.5x memory reduction for 12B parameter models while maintaining image quality

- 3.0x speedup over existing 4-bit weight-only quantization approaches

- Enables running 12B parameter models on consumer GPUs that previously couldn't handle them

And, I’ll add, as someone who has been following the diffusion space quite actively for the last two years, the amount of creativity that can be unleashed when models are accessible to people with consumer GPUs is nothing short of astonishing.

The authors took pains to validate their approach by testing it against three models (Flux, PixArt-Sigma, and SDXL) and along several quality-comparison axes (FID score, Image Reward, LPIPS, and PSNR). They also did a proper ablation study to see the contribution of each component in their approach to image quality.

What particularly excites me about this paper is not the ability to run a model that eats 22GB of VRAM in just 7GB. The exciting thing is the prospect of running a 60GB model in 20GB of VRAM. I’m not sure whether anyone has or is planning to train such a monster, but I suspect that Midjourney, OpenAI, and Google all have significantly larger models running in their infrastructure than what can be run on consumer hardware. The more dimensions you can throw at image and video generation, the better things get.

As others have replied, this is reasonable general feedback, but in this specific case the work was done carefully. Table 1 from the linked paper (https://arxiv.org/pdf/2411.05007) includes a variety of metrics, while an entire appendix is dedicated to quality comparisons.

By showing their work side-by-side with other quantization schemes, you can also see a great example of the flavor of different results you can get with these slight tweaks (e.g., ViDiT INT8) and that their quantization does a much better job in reproducing the "original" (Figure 15).

In this application, it's not strictly true that you care to have the same results, but this work does a pretty good job of it.

This is a very real concern. I've seen quantized models outputting complete garbage in LLMs. In most cases it definitely felt that a smaller unquantized model would do better. They must be included in every comparison.

E.g. compare quantized LLaMA 70B to unquantized LLaMA 8B.

Even better if the test model has a smaller version with similar byte size to the quantized larger one.

Not really. They quantized the activations here with their inference program which decreased compute as well as RAM usage (and required bandwidth). That's a big step.

Did you...did you read the technical details? This is almost all they talk about, this method was created to get around.

Take a look, it's good stuff! Basically a LoRA to reconstruct outliers lost by quantization, helping keep the performance of the original model.

It's worth noting this is laptop 4090 GPU which is more like in the range of desktop 4070 performance.

The compute differential between an H100 and a 4090 is not huge. The main single GPU benefits are larger memory (and thus memory bandwidth) and native fp8. But these matter less for diffusion models.

Damn, it runs very fast.

Hey, can you share the inference code please? Thanks..

Earlier this year there was a paper from Microsoft where they trained a 1.58 bit (every parameter being ternary) LLM that matched the performance of 16 bit models. There's also other research that you can prune up to 50% of layers with minimal loss of performance. Our current training methods are just incredibly crude and we will probably look back on those in the future and wonder how this ever worked at all.

Do you want a cookie for joining the overwhelming majority?

Necessary precision depends on, unsurprisingly, what you're truncating. Flux drops off around q6. Text generation around q4.

The llms apple are putting in iphones are q4 3b models.

I did try, but in a wrong way (try to SVD quantization error to recover quality (I.e. SVD(W - Q(W)))). The lightbulb moment in this paper is to do SVD on W and then quantize the remaining.

Diffusion requires a lot more computation to get results compared to transformers. Naively when I'm running a transformer locally I get about 30% GPU utilization, when I'm running a diffusion model I'm getting 100%.

This means that the only saving you're getting in speed for a diffusion model is being able to do more effective flops since the floats are smaller, e.g. instead of doing one 32bit multiplication, you're doing 8 4bit ones.

By comparison for transformers you not only gain the flop increase, but also the improvement in memory shuffling that they do, e.g. it also takes you 8 times less time to load the memory into working memory from vram.

The above is a vast over simplification and in practice will have more asterisks than you can shake a stick at.

GPU workloads are either compute bound (floating point operations) or memory bound (bytes being transferred across memory hierarchy.)

Quantizing in general helps with the memory bottleneck but does not help in reducing computational costs, so it’s not as useful for improving performance of diffusion models, that’s what it’s saying.

The next sentence there:

> To achieve measured speedups, both weights and activations must be quantized to the same bit width; otherwise, the lower precision is upcast during computation, negating any performance benefits.

tries to explain that.

What it means though is that if you only store the inputs in lower precision, but still upcast to say bf16 or fp32 to perform the operation, you're not getting any computational speedup. In fact, you're paying for upconverting and then downconverting afterwards.

The repo has sample code and it is fairly easy to create a node that will do it.

You won't however have access to usual sampler, latent image, Lora nodes to do anything beyond basic t2i

Why? There is nothing to customize with Flux.

They're saying this method essential does not, even when mixed with low rank models on top. "Notably, while the original BF16 model requires per-layer CPU offloading on the 16GB laptop 4090, our INT4 model fits entirely in GPU memory, resulting in a 10.1× speedup by avoiding offloading."

This is the whole magic, the rest of the workflow doesn't need to unload and flush memory, causing big delays for jobs.