KV Cache Compression and Inference Runtimes: How Innovations Like Google's TurboQuant Solve LLM Memory Bloat

Shaikhmuizz javed
Jul 28
16 min read

Ask anyone running large language models in production what actually breaks first, and it's rarely the GPU's raw compute. It's memory. Specifically, it's the kv cache — the growing pile of key and value vectors a model has to keep around in GPU memory for every single token in a conversation. Feed a modern model a 1-million-token context, the kind Gemini and Claude now support, and you can end up needing more memory just to remember the conversation than you need to store the model's entire weight file.

That's the part most explanations of "context windows" skip over. A GPU like an H100 has enormous compute throughput, but it only has so much High Bandwidth Memory (HBM), and every token you generate requires re-reading the entire history of that request from HBM. As context lengths climb into the hundreds of thousands of tokens, the kv cache stops being a minor implementation detail and starts becoming the single biggest constraint on how many users a company can actually serve on a given GPU fleet.

In March 2026, Google Research published TurboQuant, an algorithm that compresses this cache down to roughly 3 bits per value with what the paper describes as near-zero downstream accuracy loss — a claim that's now been independently reproduced in early open-source ports to vLLM and llama.cpp. This article breaks down why the kv cache became the bottleneck in the first place, how compression techniques evolved to address it, and what TurboQuant specifically does differently from the quantization methods that came before it.

AI infographic on KV cache compression and inference runtimes, showing smarter memory, faster AI, TurboQuant, and performance gains.

What Is the KV Cache in LLM Inference?

The autoregressive token generation cycle

Transformer-based language models generate text one token at a time. To produce token number 500, the model needs to run self-attention across all 499 tokens that came before it — comparing the new token's "query" against the "keys" of every prior token, then using those comparison scores to pull relevant information out of the corresponding "values."

Here's the redundancy problem: if you recomputed the keys and values for tokens 1 through 499 from scratch every single time you generated a new token, you'd be repeating the exact same matrix multiplication work over and over, layer after layer, for a result that never changes. The keys and values for a token, once computed, stay fixed for the rest of that generation.

So inference engines cache them. The kv cache stores the key and value vectors for every token, at every attention head, in every transformer layer, the moment they're first computed. Generating token 500 becomes a matter of computing one new query vector and comparing it against the cache — not recomputing history. This is what makes autoregressive generation tractable at all. It's also what quietly turns into a memory problem the moment context length grows.

Why context-window memory scales linearly

The size of the kv cache follows a fairly simple formula:

Memory (KV Cache) = 2 × B × L × H × D × S × P

Where:

B = batch size (how many requests you're serving concurrently)
L = number of transformer layers
H = number of attention heads
D = dimension size per head
S = sequence length (context tokens processed so far)
P = precision in bytes (2 bytes for FP16/BF16)
The leading 2 accounts for storing both keys and values

Plug in realistic numbers for a mid-sized open model — say 80 layers, 64 heads, a head dimension of 128 — and a single request sitting at a 128,000-token context can require somewhere in the neighborhood of 20–32GB of VRAM just to hold the cache. That's before you've loaded a single model weight. For context, that's comparable to or larger than the entire weight footprint of many mid-size open-source models. This is why "long context" marketing numbers from vendors always come with an asterisk about serving cost — the context window isn't just a prompt-size limit, it's a direct multiplier on GPU memory consumption per user.

The memory vs compute bottleneck: prefill vs decode

Inference happens in two distinct phases, and they behave completely differently on the hardware.

Prefill is when the model processes your entire input prompt for the first time. This is compute-bound — it's one big, highly parallelizable matrix multiplication across every prompt token at once, and modern GPUs are extremely good at that kind of dense, parallel math.

Decode is different. Once generation starts, the model produces one token at a time, and for each new token it has to read the entire kv cache — every key and value for every prior token — out of HBM and into the much faster on-chip SRAM to run attention. This phase is memory-bandwidth-bound, not compute-bound. The GPU's math units sit relatively idle while it waits for data to move across the memory bus.

This distinction matters because it explains why bigger, faster GPUs don't automatically fix long-context serving. You can have a chip with enormous FLOPs and still be throttled during decode, because the limiting factor isn't how fast you can multiply matrices — it's how fast you can move an ever-growing cache from HBM to SRAM, once per generated token, for the entire duration of the conversation.

The Mechanics of KV Cache Compression

Sparse attention and eviction policies

The earliest fixes to this problem didn't compress the cache at all — they just threw parts of it away. Approaches like H2O (Heavy Hitter Oracle) and StreamingLLM work on the observation that not every past token matters equally to future predictions. H2O tracks which tokens accumulate the highest attention scores over time ("heavy hitters") and evicts the rest, keeping the cache size roughly constant instead of letting it grow with every new token. StreamingLLM takes a related but simpler approach, permanently anchoring a small number of initial "sink" tokens plus a sliding window of recent tokens, and discarding everything in between.

Both methods work reasonably well for tasks that lean on recent or locally important context. Where they struggle is mid-context retrieval — the classic "needle in a haystack" scenario, where a critical fact is buried somewhere in the middle of a long document and the model needs to recall it precisely later. If an eviction policy has already dropped that token, the information is gone, not just degraded. That's a real accuracy trade-off, not a rounding error, and it's the main reason eviction-based methods never fully solved the problem — they solved a different problem, which was raw cache size, at the cost of recall reliability.

Standard quantization (INT8, FP8, INT4) and the outlier challenge

The alternative to throwing data away is compressing it. Standard quantization takes each cached value, normally stored as a 16-bit float, and represents it with fewer bits — 8-bit, or more aggressively, 4-bit integers. Unlike eviction, nothing is deleted; every token's information is retained, just at lower numeric precision.

This works cleanly for weight quantization, but the kv cache resists it for a specific reason: activation outliers. In real transformer activations, a small number of channels regularly produce values with much larger magnitude than the rest — sometimes by an order of magnitude or more. A standard scalar quantizer has to size its numeric range around the largest value in a block, which means the outlier channels dominate, and every other, smaller value gets crushed into a handful of representable levels. At 8-bit precision this is a manageable annoyance. At 3-bit or 4-bit, it becomes a serious accuracy problem — the compression ratio looks great on paper, but downstream generation quality falls off a cliff, especially on tasks that depend on precise recall rather than general fluency.

Block-wise quantization vs online quantization

One partial fix is block-wise quantization — grouping nearby values together and computing a separate scale factor per block, rather than one scale for an entire tensor. This narrows the damage outliers can do, but it comes in two flavors with very different practical behavior.

Static, offline-calibrated quantization picks its scale factors ahead of time using a representative calibration dataset, then applies those fixed parameters at inference. It works well as long as production traffic resembles the calibration data — and degrades noticeably when it doesn't, which is a real risk for conversational and agentic workloads where prompt distributions shift constantly.

Online, per-token quantization recomputes scale factors on the fly as each token is cached, adapting to whatever distribution is actually showing up in production. It's more robust to distribution shift, but the recalculation has to happen fast enough that it doesn't eat into the latency savings the compression was supposed to deliver in the first place. This tension — accuracy versus speed, at every single token — is exactly the gap TurboQuant was built to close.

Inside Google's TurboQuant: Redefining KV Cache Quantization

Google Research introduced TurboQuant in a research paper on arXiv in April 2025, with the full system slated for presentation at ICLR 2026 and its companion technique, PolarQuant, presented separately at AISTATS 2026. Google published an accompanying technical breakdown on its Google Research blog, and the method has since been ported into early experimental builds for vLLM, llama.cpp, and Apple's MLX framework.

Infographic titled Solving LLM Memory Bloat: The Power of TurboQuant, showing KV cache crisis, GPU, and 6x memory reduction charts.

The PolarQuant method: random rotation to combat vector outliers

TurboQuant's first move is deceptively simple: before quantizing anything, it multiplies each key or value vector by a random orthogonal matrix. An orthogonal transformation rotates a vector in high-dimensional space without stretching, shrinking, or distorting it — the vector's length and its geometric relationship to every other vector stay exactly the same.

What changes is where the energy sits. In the original, un-rotated vector, a handful of outlier channels might carry most of the magnitude, while the rest sit near zero. After a random rotation, that same total energy gets spread roughly evenly across all coordinates. No single dimension dominates anymore, which means a scalar quantizer no longer has to size its numeric range around a handful of extreme values — it can compress every coordinate with a consistent, tight range. This is the technique published as its own companion paper, PolarQuant, and it's the reason TurboQuant can push down to 3-bit and even lower without the outlier collapse that sinks standard INT4 quantization.

Quantized Johnson-Lindenstrauss (QJL) transform for bias correction

Rotation alone doesn't fully solve the problem — any quantizer still introduces some reconstruction error, and if that error is systematically biased in one direction, it distorts the attention math downstream, even if the raw numbers look "close enough" by a mean-squared-error measure.

TurboQuant's second stage addresses this directly with a technique called Quantized Johnson-Lindenstrauss (QJL), derived from the classical Johnson-Lindenstrauss lemma in dimensionality reduction theory. Rather than storing the residual error in full, TurboQuant computes a compact 1-bit sketch of it and uses that sketch to correct the inner-product estimate at attention time. The result is a quantizer whose output isn't just numerically close to the original — it's statistically unbiased for the specific operation that matters, which is the query-key inner product that drives attention scores. That distinction between "low reconstruction error" and "unbiased inner-product estimation" is the core theoretical contribution separating TurboQuant from earlier compression schemes.

Data-oblivious compression: zero retraining, zero calibration data

Because the rotation matrices are randomly generated rather than learned, and the quantization thresholds are computed mathematically rather than fitted to a dataset, TurboQuant is what the paper calls data-oblivious. It requires no calibration set, no fine-tuning pass, and no per-model training run. You apply it to an already-trained model's kv cache at inference time and it works immediately — a meaningful practical advantage over codebook-based vector quantization methods like AWQ or GPTQ, which need representative data to optimize their codebooks before deployment. Independent reports suggest engineers were porting TurboQuant into frameworks like MLX within days of the paper's release, which is a reasonable indicator of how low the integration barrier actually is.

Performance gains: 6x memory reduction and up to 8x faster H100 attention

According to Google's published benchmarks, TurboQuant compresses the kv cache by more than 6x relative to standard 16-bit storage, while scoring perfectly on RULER — a needle-in-haystack benchmark specifically designed to catch the kind of mid-context retrieval failures that plague eviction-based methods. On NVIDIA H100 GPUs, 4-bit TurboQuant delivered up to an 8x speedup in computing attention logits compared to unquantized 32-bit keys, by shrinking the volume of data that has to move from HBM into SRAM during the decode phase — directly attacking the memory-bandwidth bottleneck described earlier.

It's worth being precise here: the 6x memory figure and the 8x speed figure come from Google's own reported benchmarks on their test setup, and early independent implementations (including community ports to vLLM and llama.cpp) have reported broadly consistent, if somewhat variable, results depending on model architecture, bit-width configuration, and whether fused Triton kernels are used versus a simpler dequantize-then-attend path.

Modern Inference Runtimes and KV Cache Architecture

Compression algorithms don't run in a vacuum — they need serving infrastructure built to actually exploit them.

PagedAttention and vLLM: virtual memory allocation

Before vLLM, most inference engines allocated kv cache memory in large, contiguous blocks sized for the worst case — a request's maximum possible sequence length — which wasted enormous amounts of VRAM to fragmentation, since most requests never used their full allocation. vLLM's PagedAttention borrowed a decades-old idea from operating systems: treat GPU memory like virtual memory, breaking the kv cache into fixed-size, non-contiguous "pages" that get allocated on demand and mapped through a page table. This alone dramatically improved GPU memory utilization and became a foundational piece of modern LLM serving, independent of whatever quantization scheme sits on top of it. You can find the project on vLLM's GitHub repository.

TensorRT-LLM and Triton: dynamic split-k and flash decoding

NVIDIA's TensorRT-LLM, along with custom kernels written in Triton, take a different but complementary angle: making the actual attention computation faster on the hardware. FlashDecoding specifically parallelizes the decode phase's attention calculation across multiple GPU thread blocks rather than processing the full cache sequentially, which helps offset the memory-bandwidth ceiling by extracting more parallelism from the read operation itself. Techniques like these are what make compressed caches practically useful — a 6x smaller cache is only as valuable as the kernel's ability to actually read and unpack it fast during inference.

Apple MLX and local inference optimization

On the consumer and edge side, Apple's MLX framework benefits from these same ideas in a slightly different context. Apple Silicon uses a unified memory architecture, where CPU and GPU share the same physical memory pool rather than the GPU having its own dedicated VRAM. That shared pool is typically much smaller than a datacenter GPU's HBM, so kv cache compression schemes have an outsized impact locally — they're often the difference between a 32k-token context fitting comfortably on a consumer Mac and not fitting at all.

Comparison Matrix: KV Cache Optimization Strategies

Here's how the major approaches stack up against each other across the dimensions that actually matter for production deployment: memory savings, accuracy impact, and how much extra compute overhead each method introduces.

Compression Strategy	Bit-Width	Memory Savings	Downstream Accuracy Impact	Computational Overhead	Primary Hardware Fit
FP16 / BF16 (Baseline)	16-bit	1x (None)	None (Baseline)	Low (no unpacking)	All GPUs
FP8 / INT8 Quantization	8-bit	2x	Extremely low	Low (hardware-supported)	H100, RTX 4090, Hopper+
Standard INT4 Quantization	4-bit	4x	Moderate (outlier clipping loss)	Low–Medium	Hopper / Ada Lovelace
Token Eviction (StreamingLLM / H2O)	Mixed	Variable (bounds active context)	High risk on mid-context retrieval	Minimal	General CPUs/GPUs
Google TurboQuant	3-bit Key / 2-bit–4-bit Value	~6x	Near-zero (unbiased residual correction)	Medium (rotation + QJL steps)	Modern Tensor Cores / Triton kernels

A quick reading note on that last row: independent community benchmarks have flagged that value quantization is generally the more sensitive of the two — aggressive 2-bit value compression can show measurable similarity degradation on some workloads, with 4-bit values often preferred where output quality is critical. Key compression down to 3-bit tends to hold up more reliably. This is a useful reminder that "6x compression" is a headline number, and the actual configuration you'd run in production depends on how quality-sensitive your specific use case is.

Enterprise and Infrastructure Impact of Low-Bit KV Caching

Driving down serving costs and HBM requirements

The direct consequence of shrinking the kv cache is straightforward: a request that used to need 32GB of VRAM just for its context memory might need closer to 5–6GB after compression. That freed-up memory doesn't just sit idle — it lets a cloud provider or enterprise inference team pack more concurrent requests onto the same GPU, which is where the real cost savings show up. Since inference now represents the large majority of ongoing spend for most companies running AI at scale, anything that increases effective throughput per GPU has a direct line to the bottom line, independent of whatever the model itself costs to train.

Jevons paradox: why cheaper context scales application design

There's an economic wrinkle worth flagging here. When the cost of a resource drops, consumption of that resource doesn't shrink proportionally — it tends to grow, because the lower cost unlocks use cases that weren't previously viable. This is the classic Jevons paradox, and it applies neatly to context windows. If serving long context gets 6x cheaper, most organizations won't pocket a 6x savings on their inference bill. Instead, developers will start feeding entire codebases, full document histories, and multi-modal conversation logs directly into prompt buffers that would have been prohibitively expensive to serve before. Cheaper context doesn't just optimize existing applications — it changes what people build in the first place.

Tiered caching strategies: hot, warm, and cold tokens

At the infrastructure level, this is pushing serving architectures toward tiered memory design, similar to how databases handle hot and cold storage. Recently accessed, "hot" tokens stay in the active GPU kv cache for fast attention access. Older, less frequently referenced "warm" or "cold" tokens can be offloaded to cheaper system DRAM or even NVMe storage, then decompressed and loaded back into the GPU's kv cache on demand when a request actually needs them. Combined with a compression scheme like TurboQuant, this kind of tiering lets platforms support extremely long effective context histories without permanently reserving premium HBM capacity for information that's rarely touched.

Risks, Limitations, and Implementation Challenges

No compression technique is free, and it's worth being direct about where the friction actually sits.

Accumulation of quantization noise in deep layers. Modern models routinely run 80 or more transformer layers. Even a small amount of quantization error introduced at each layer can compound as it propagates forward, and while TurboQuant's unbiased correction is specifically designed to prevent systematic drift, extremely precision-sensitive tasks — multi-step mathematical reasoning or exact code execution, for instance — are worth extra scrutiny before deploying aggressive compression in production, rather than assuming benchmark numbers transfer perfectly to every workload.

Integration constraints in non-transformer architectures. State-space models like Mamba and RWKV don't use a traditional kv cache at all — they compress history into a fixed-size recurrent state that updates as new tokens arrive, rather than storing every past token explicitly. That's a fundamentally different memory model, and it means TurboQuant, which is built around compressing discrete key and value vectors, isn't directly applicable to these linear-state architectures. It's a technique for transformer-style attention specifically, not a universal fix for sequence-model memory.

Kernel-level compilation and hardware portability hurdles. Getting the theoretical compression ratio to translate into actual wall-clock speedup requires writing custom CUDA or Triton kernels that can perform the rotation, quantization, and unpacking steps fast enough that they don't eat the savings they're supposed to deliver. This is nontrivial engineering work, and it's part of why early community implementations flag things like "hybrid decode paths that dequantize all history" as a known limitation still being optimized — the theory arrived before the fully mature production kernels did.

Future of Long-Context Inference

On-device reasoning and local extreme context. As compression techniques mature and get baked into frameworks like MLX and llama.cpp, running genuinely long context windows — 100k tokens and beyond — on laptops and mobile hardware, without a cloud round-trip, is becoming realistic rather than aspirational. That has real implications for privacy-sensitive applications and offline AI tooling.

Linear-attention hybrids and the end of dense caching. A parallel trend worth watching is architectural: some newer model designs are moving toward linear-attention or hybrid state-space components specifically to sidestep the kv cache growth problem at the source, rather than compressing it after the fact. Compression and architectural redesign aren't competing solutions — they're two different levers on the same underlying cost, and it's likely both continue to develop in parallel rather than one simply replacing the other.

Conclusion

The story of LLM scalability in 2026 isn't really about bigger models anymore — it's about memory efficiency. The kv cache quietly became the real ceiling on how much context a system can serve and how many users it can support per GPU, and for a while, the tools available to address it forced an uncomfortable trade-off between throwing data away and accepting real accuracy loss from aggressive quantization.

TurboQuant is a useful case study in how that trade-off gets broken. It doesn't rely on a bigger GPU or a smarter eviction heuristic — it relies on genuine mathematical insight: rotate vectors to neutralize outliers,

then apply a bias-corrected residual sketch to keep the compressed cache statistically faithful to the original. That's a meaningful reminder that some of the biggest efficiency gains in AI infrastructure right now aren't coming from new hardware. They're coming from smarter math applied to the hardware we already have.

If your team is evaluating long-context deployment, inference cost optimization, or broader AI infrastructure strategy, FourfoldAI's technical consulting works directly with enterprise teams to translate research like this into practical serving decisions. Get in touch to talk through your architecture.

KV Cache Optimization: Frequently Asked Questions

What is the KV cache?

The kv cache (Key-Value cache) is a memory allocation technique used during autoregressive Large Language Model inference that stores the key and value vectors of previously processed tokens so they don't need to be recalculated when generating each new word, reducing computational latency.

What is KV cache compression?

KV cache compression refers to strategies — including token eviction, pruning, and low-bit quantization — designed to reduce the physical VRAM footprint occupied by saved key and value states during long-context LLM generation without significantly compromising accuracy.

What is Google's TurboQuant?

Google's TurboQuant is an online, training-free vector quantization algorithm that compresses an LLM's kv cache down to roughly 3-bit representations. It preserves downstream accuracy through random orthogonal rotations (PolarQuant) combined with bias-corrected residual calculations (QJL).

How does TurboQuant achieve near-zero accuracy loss?

TurboQuant uses the PolarQuant method to randomly rotate vector coordinates, spreading outlier energy evenly and making values easier to compress uniformly. It then applies a 1-bit Quantized Johnson-Lindenstrauss (QJL) transform to mathematically correct residual quantization errors, keeping attention-score estimates statistically unbiased.

Why are AI inference costs bound by VRAM memory?

AI inference costs are bound by VRAM memory because the decode phase of generation requires reading the entire historical kv cache from GPU High Bandwidth Memory (HBM) for every new token. As context windows scale, this cache consumes massive memory bandwidth and capacity, often becoming a bigger constraint than raw compute.

How do teacher and student AI models work in distillation?

In knowledge distillation, a smaller "student" model is trained to mimic the outputs of a larger "teacher" model — typically by matching the teacher's output probability distributions (logits) rather than just the final predicted answer, which transfers more nuanced information about how the teacher weighs different possibilities. Some approaches also align intermediate attention patterns between the two models.

What is the difference between AI model compression and distillation?

Model compression techniques — pruning, quantization, and kv cache compression among them — reduce the memory or compute footprint of an existing model's weights or activations. Distillation, by contrast, trains an entirely new, smaller model architecture from scratch, using a larger model's outputs as its training signal.

Is model distillation legal?

The legality of distillation is genuinely unsettled and depends heavily on jurisdiction, the specific terms of service of the model being distilled from, and how the training data was obtained. Several providers explicitly restrict using their model outputs to train competing models in their usage policies, and this remains an active area of legal and industry debate rather than a settled question.

References and Further Reading

This article draws on the following primary and technical sources:

Zandieh et al., "TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate," arXiv, 2025 (ICLR 2026)
Google Research, "TurboQuant: Redefining AI efficiency with extreme compression"
vLLM GitHub repository — PagedAttention and open-source serving infrastructure

Related reading on FourfoldAI: Small Language Models (SLMs) | AI Cost Optimization | AI Infrastructure

Disclaimer:

This article is intended for informational and educational purposes only. While every effort has been made to ensure technical accuracy based on publicly available research and documentation at the time of writing, AI infrastructure techniques evolve rapidly, and readers should verify current specifications before making architecture or deployment decisions. For full details, see our disclaimer page.

About the Author

Muizz Shaikh is an AI enthusiast and digital technology professional at FourfoldAI. He is passionate about exploring AI tools, industry trends, and practical applications of emerging technologies. Through FourfoldAI, Muizz contributes to simplifying artificial intelligence for businesses and learners. Connect with him on LinkedIn: linkedin.com/in/muizz-shaikh-45b449403/