NVMe KV Cache Offloading for vLLM and Ollama in 2026: The Honest Long-Context Setup Guide

By AIFoss · 9 min read
vllmollamakv-cacheselfhostedllamacpp

TL;DR: NVMe KV cache offloading is real and works today — but only through vLLM paired with LMCache (both Apache 2.0/MIT-family FOSS). Ollama has no disk-KV path as of the v0.32.x line, and llama.cpp offers save/restore snapshots rather than live tiering. And no, none of this lets a consumer GPU run Kimi K3: offloading moves the cache, not the 600GB+ of weights.

What you’ll have running after this guide:

  • vLLM serving a local model with LMCache tiering KV blocks across GPU → CPU RAM → NVMe, so long prefixes survive restarts and don’t get recomputed
  • A KV cache config you can size correctly because you understand the per-token math
  • A realistic picture of what Ollama and llama.cpp can and can’t do here, so you pick the right runner instead of fighting the wrong one

Honest take: If your workload is multi-turn RAG, agent loops, or repeated long documents, vLLM + LMCache with a fast NVMe tier is genuinely worth an afternoon of setup. For interactive chat on a single GPU, skip it — KV quantization in Ollama gets you most of the win with none of the plumbing.

First, the correction: this does not put Kimi K3 on your RTX 4090

The posts that made this technique trend paired it with Kimi K3’s 1M-token context window, implying a consumer GPU could now run Moonshot’s 2.8T-parameter model if it just had a big enough SSD. That’s wrong, and it’s worth being precise about why.

KV cache offloading moves the attention cache — the per-token key/value tensors a model accumulates as context grows — off the GPU. It does nothing about the weights. Kimi K3’s weights need roughly 610GB of combined RAM and VRAM even at the most aggressive 1-bit Unsloth quantization, and mainline llama.cpp still doesn’t support its KDA attention architecture. We covered the full hardware reality in our Kimi K3 self-hosting guide; the verdict — use the API, or rent datacenter GPUs — is unchanged by anything on this page.

What NVMe KV offloading actually buys you: a model that already fits your GPU can serve far longer contexts, more concurrent sessions, and warm-restart without recomputing every cached prefix. That’s a real and useful win. It’s just a different win than the viral framing promised.

The math: why KV cache becomes the problem

For a standard transformer, KV cache size scales linearly with context length:

KV bytes ≈ 2 × layers × kv_heads × head_dim × bytes_per_param × tokens

Take a typical 32-layer, 8-KV-head, 128-head-dim model at FP16: that’s roughly 128KB per token — about 16GB of KV cache at 128K context, on top of the weights. Push toward very long contexts and the cache rivals or exceeds the model itself. On a 24GB card running a ~17GB Q4 model, you have ~7GB left for KV: the cache, not the model, is what caps your usable context.

Three FOSS responses exist, in increasing order of ambition:

  1. Quantize the KV cache (q8_0 halves it, q4_0 quarters it) — supported by llama.cpp, Ollama, and vLLM.
  2. Offload cache to CPU RAM — vLLM natively, llama.cpp partially (--no-kv-offload keeps KV in system RAM).
  3. Tier cache to NVMe so it persists and scales past RAM — this is LMCache territory, and vLLM is the only mainstream runner wired for it.

The setup that works: vLLM + LMCache

LMCache is an Apache 2.0 KV cache engine that plugs into vLLM as a KV connector. Per its README it tiers cache across CPU RAM, local disk (SSD/NVMe), and remote backends (Redis/Valkey, Mooncake, S3-compatible stores). The critical difference from vLLM’s built-in --swap-space: swap space is scratch memory that vanishes on restart, while LMCache’s disk tier persists — a server restart doesn’t force re-prefilling every warm prefix.

Install

pip install vllm lmcache

Use a recent vLLM (the LMCache connector targets the V1 engine; we wrote this against the v0.26.x line current in mid-2026). Match CUDA versions between the two packages — mismatches fail at import time, not silently.

Configure the tiers

Create lmcache.yaml:

chunk_size: 256
local_cpu: true
max_local_cpu_size: 16        # GB of system RAM for the hot tier
local_disk: "file:///mnt/nvme/lmcache/"
max_local_disk_size: 200      # GB of NVMe for the cold tier

Point local_disk at a mount on your fastest SSD. This is a workload where the drive genuinely matters: KV blocks stream back during prefix restore, so a PCIe 4.0 drive like a Samsung 990 Pro 2TB (~7GB/s reads) keeps restore latency tolerable where a SATA SSD (~550MB/s) makes the whole exercise pointless. Our sister site ran the hardware numbers in its NVMe KV cache buying guide — short version: read throughput and sustained-write behavior matter more than capacity.

Launch vLLM with the connector

LMCACHE_CONFIG_FILE=./lmcache.yaml \
vllm serve Qwen/Qwen3.8-27B-Instruct \
  --kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}' \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.90

Recent vLLM releases also expose a shortcut — --kv-offloading-backend lmcache with --kv-offloading-size <GiB> — that skips the JSON config for simple CPU-tier setups. Check vllm serve --help on your installed version; the kv-transfer-config path above is the one that’s been stable across releases.

Verify it’s active: startup logs should show Initializing LMCacheConfig under kv_transfer_config. Then send the same long prompt twice — the second run’s prefill time should collapse, and after a full server restart it should stay collapsed. That last check is the one --swap-space can’t pass.

If you’re renting rather than owning the GPU, this stack works identically on a RunPod pod — attach a network volume for the LMCache disk tier so the cache survives pod restarts too.

Ollama: not supported, and here’s what to do instead

Ollama has no KV-to-disk capability as of the v0.32.x line. We checked the tracker rather than guessing: the closest open request is issue #9750, which asks Ollama to prefer keeping KV in system RAM over dropping model layers when both don’t fit — one user reported a QwQ-32B run at 131K context on a 16GB card going from ~2 hours of thinking to ~35 minutes with the underlying llama.cpp --no-kv-offload behavior. The maintainers’ current VRAM split does the reverse, and nothing on the roadmap touches NVMe tiering.

What Ollama does give you is KV quantization:

OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve

q8_0 halves KV memory with negligible quality loss (flash attention required, and note some architectures have had quantization bugs — check the tracker for your model family). That often doubles your usable context on the same card, which for a solo chat workload is the 80% solution. If you genuinely need tiered, persistent, past-RAM KV storage, the honest answer is to run vLLM (or llama-server) underneath instead of waiting for Ollama to grow the feature.

llama.cpp: snapshots, not tiering

llama.cpp gets misdescribed in this discussion, so let’s be precise about what it has:

  • KV quantization via --cache-type-k q8_0 --cache-type-v q8_0 — same win as Ollama’s env vars (Ollama inherits this from llama.cpp).
  • State save/restore: llama-server can persist a slot’s KV state to disk (--slot-save-path plus the slot save/restore API), and llama-cli has --prompt-cache. This is a snapshot you explicitly save and reload — great for reusing one big system prompt or document prefix, but it’s not automatic tiering that pages blocks in and out during generation.
  • mmap applies to weights, not KV. A memory-mapped model file is not KV offloading, whatever the Reddit thread said.

For a single recurring mega-prefix (say, a 200K-token codebase dump you query daily), the snapshot approach is actually the simplest FOSS path of all three. For many varied long prefixes, it doesn’t scale — that’s LMCache’s job.

Performance reality: when the trade-off is worth it

Restoring KV from NVMe is much faster than recomputing prefill on long prompts — that’s the entire value proposition — but a cache miss or cold restore still adds seconds of latency, and heavy paging costs generation throughput versus pure-GPU KV. We didn’t have a GPU rig in this writing environment to publish our own tok/s deltas, so treat vendor throughput claims (LMCache cites large TTFT wins on multi-turn workloads) as directional and benchmark your own stack.

The workload split is what actually decides it:

  • Worth it: multi-turn RAG over shared documents, overnight agent runs, batch summarization, multi-user servers with overlapping prefixes, anything where restarts currently torch a warm cache. Pair it with a proper serving setup — see our vLLM production guide and multi-GPU setup.
  • Not worth it: interactive single-user chat, short contexts, models that already fit comfortably. Start with KV quantization and our context window guide — most people need less context than they think, and a used RTX 3090 with 24GB of VRAM solves more long-context problems than any SSD does.

FAQ

Does NVMe KV offloading let me run bigger models? No. It offloads the attention cache, not the weights. Model size is bounded by VRAM + RAM (and quantization) exactly as before. It lets a model that already fits serve longer contexts and more sessions.

Will offloading wear out my SSD? KV tiering is write-heavy, but modern TLC drives are rated for hundreds of TB written per year of this kind of duty. A 2TB 990 Pro carries a 1,200 TBW rating; a hobbyist workload won’t approach it. Avoid QLC drives with small SLC caches — sustained-write collapse hurts here.

Can I share the NVMe cache tier between multiple vLLM instances? On one machine, yes — LMCache instances can point at shared storage. Across nodes, LMCache supports remote backends (Redis/Valkey, S3-compatible, Mooncake) designed exactly for multi-replica cache sharing.

  • Samsung 990 Pro 2TB — PCIe 4.0 NVMe with the sustained read throughput a KV disk tier needs
  • RTX 3090 24GB — still the used-market VRAM king; more VRAM beats cleverer offloading

Sources

Was this article helpful?