Is Q4_K_M good enough for everyday use?

For most chat and coding work, yes. A January 2026 study of Llama 3.1 8B Instruct found Q4_K_M averaged 69.15 across a benchmark suite against 69.47 at 16-bit, with perplexity rising from 7.32 to 7.56. The loss becomes more noticeable at Q3 and below, and maths tasks are hit hardest.

What is the difference between Q4_K_M and Q4_K_S?

Both are 4-bit k-quants from llama.cpp. The M version stores half of the attention value tensors and half of the feed-forward down-projection tensors at 6-bit instead of 4-bit, so it is slightly larger and slightly more accurate than the S version. Q4_K_M is the default that most tools pick.

Should I use Q8_0 or Q4_K_M?

Use Q8_0 when the model fits with room for context, since it is close to lossless: it scored 69.41 against 69.47 at 16-bit in the January 2026 study. If Q8_0 does not fit, a bigger model at Q4_K_M usually beats a smaller one at Q8_0, and Q4_K_M also generates tokens faster.

What are I-quants like IQ4_XS?

I-quants are llama.cpp formats built for low bit rates that work best with an importance matrix made from calibration text. IQ4_XS averages about 4.46 bits per weight against 4.89 for Q4_K_M. The IQ2 and IQ1 levels make files of about 2 to 2.4 bits per weight possible, at a real quality cost.

Does quantizing the KV cache hurt quality?

An 8-bit KV cache halves context memory and usually has no noticeable impact on quality, according to the Ollama documentation. A 4-bit cache uses a quarter of the memory but the loss can become noticeable at long context, so test it on your own work first.

GGUF Quantization Explained: Q4_K_M, Q8_0 and the Size Math