TL;DR: Four open-source stacks claim to make CPUs useful for LLM inference in 2026, and they are not interchangeable. llama.cpp (and the ik_llama.cpp fork) is the only genuinely CPU-only option worth running. KTransformers v0.7.0 is the best tool for giant MoE models but requires a GPU alongside the CPU — it is hybrid, not CPU-only. vLLM’s x86 CPU backend works and supports AWQ/GPTQ, but it targets batch serving, not a single chat session. PowerInfer is research-grade with fading maintenance. And the 4x DDR5 price spike means the old “RAM is cheap, VRAM is not” argument that justified CPU inference needs re-checking before you spend anything.

Why is CPU inference back in 2026?

Two reasons: sparse MoE models and memory-bandwidth math — but the economics underneath them shifted this year.

Modern open-weight MoE models activate only a few billion parameters per token. A 26B-class MoE with ~4B active parameters needs each token to read roughly 2-3 GB of weights at Q4 quantization, not 15 GB. Dual-channel DDR5 moves about 80-100 GB/s, so the arithmetic predicts usable single-digit tok/s on a plain desktop — and real measurements agree. The widely circulated June 2026 Neomind Labs result ran Gemma 4 26B (MoE, ~4B active) at 5.2 tok/s on a 13-year-old Xeon with DDR3 and no GPU at all. CPU inference in 2026 is bandwidth-bound, not compute-bound; the software’s job is to keep the cores efficient enough to hit the bandwidth ceiling. Our ik_llama.cpp review walks through that math in detail.

The caveat nobody puts in the headline: the DRAM price crisis repriced this whole strategy. As of September 2026, a 64GB DDR5 kit runs $680-$1,070 and 32GB runs $399-$479 (Tom’s Hardware RAM price index, verified 2026-09) — roughly 4x the mid-2025 prices, because Samsung, SK Hynix, and Micron shifted capacity to HBM. “Load up on cheap RAM instead of buying a GPU” was true in 2025. In late 2026, 128GB of DDR5 can cost more than a used RTX 3090. Keep that number in mind through everything below.

What does KTransformers actually do?

KTransformers (github.com/kvcache-ai/ktransformers, Apache 2.0, v0.7.0 released August 17, 2026) is a CPU-GPU heterogeneous inference framework: it keeps the hot path of a huge MoE model on a GPU and streams the expert layers through optimized CPU kernels — Intel AMX acceleration for INT4/INT8, AVX-512 and AVX2 dispatch, and NUMA-aware memory placement for multi-socket servers.

The honest framing: KTransformers is not a CPU-only stack. Every supported configuration in the project’s documentation pairs a CPU with at least one 24GB GPU. What it buys you is the ability to run models that are otherwise API-only — DeepSeek V3/R1/V4-Flash, Kimi K2.x, GLM-5 series, Qwen3 MoE, MiniMax M2/M3 — on a workstation where the GPU alone could never hold them. The project’s published figure for DeepSeek-R1-0528 at FP8 is 227.85 tok/s total throughput (87.58 tok/s output) on a server pairing 8x L20 GPUs with a Xeon Gold 6454S — that is project-reported, data-center-class hardware, and not what a home lab will see. Community home-lab reports cluster around a different shape entirely: one consumer GPU plus 256-512GB of system RAM running a 400B-class MoE at interactive-but-not-fast speeds.

Who it is for: the person who wants frontier-class open weights (not a 27B distill) and already owns a 24GB card plus a workstation or EPYC board that can take a lot of RAM. At September 2026 DDR5 prices, assembling 256GB+ from scratch is a four-figure line item before you buy anything else — price the RAM first.

Maintenance is healthy: 1,300+ commits, September 2026 updates added Kimi K2.5/K2.6 fine-tuning support via the LlamaFactory integration.

Is vLLM’s CPU backend production-ready?

Yes for batch serving on the right CPU, no as a desktop chat engine. vLLM ships a supported x86 CPU backend with FP32, FP16, and BF16 data types; the docs recommend the avx512f CPU flag, with AVX2 supported but explicitly limited. On x86 CPUs it accepts AWQ and GPTQ quantized checkpoints plus llm-compressor INT8 (W8A8) — note that is HuggingFace-format quantization, not GGUF. AMD is pushing the same direction: its PACE framework for EPYC CPU inference shipped as a vLLM plugin in 2026.

What the CPU backend is actually for is the same thing vLLM itself is for: continuous batching and OpenAI-compatible serving. If you have a 32-core AVX-512 server and a trickle of concurrent requests — internal RAG, a team chatbot, overnight batch classification — vLLM on CPU gives you the same API and scheduler you would run on a GPU, with no CUDA dependency. For a single interactive user it is the wrong tool: there is no GGUF support, model weights load in 16-bit unless you find an AWQ/GPTQ export, and single-stream latency loses to llama.cpp on the same silicon. Our Ollama vs vLLM comparison covers the same single-user-vs-serving split on GPU.

Before picking vLLM CPU (or ik_llama.cpp’s AVX-512 kernels), check what your silicon actually has — this decides the whole stack:

$ lscpu | grep -oE 'avx512f|avx2' | sort -u
avx2
avx512f

If avx512f is missing (most Intel desktop chips from 12th-14th gen have it fused off; AMD Zen 4/Zen 5 have it), vLLM’s CPU backend runs in its limited AVX2 mode and loses most of its appeal — llama.cpp handles AVX2-only hardware more gracefully.

Setup is also still the least friendly of the four: you build from source or use the project’s Dockerfile.cpu, and tuning (KV cache space, core pinning) is on you.

Has mainline llama.cpp caught up with ik_llama.cpp?

Not on CPU token generation. The gap that made the fork famous is still visible in 2026: on the same Xeon E5-2683 v4, the fork’s own comparison threads show roughly 5.05 tok/s versus mainline’s 2.70 tok/s for quantized generation — about 2x, and independent reports agree on direction. Mainline moves faster on ecosystem breadth (Vulkan, Metal, ROCm, SYCL, day-one model support); the fork, maintained by the developer who wrote most of mainline’s quantization types, stays ahead on CPU kernels and quant quality (AVX2 minimum, AVX-512 preferred).

For a CPU-only box, the practical ranking has not changed since August: ik_llama.cpp if you will build from source, mainline llama.cpp (or Ollama on top of it) if you want zero friction and can accept roughly half the speed. Both read standard GGUF files, which is the quantization ecosystem with actual depth — Q4_K_M for everything, published the day a model drops.

Is PowerInfer still worth watching?

Watching, yes. Deploying, no. PowerInfer (github.com/SJTU-IPADS/PowerInfer, MIT) pioneered sparse-activation offloading — predicting which neurons fire and keeping the hot ones on fast memory. The repository’s last push was May 11, 2026, there are still no tagged releases, and recent activity has centered on the group’s SmallThinker models rather than the engine itself. It supports a narrow model list and nothing from the 2026 MoE wave that KTransformers serves. The ideas won — every hybrid stack borrows them — but as a tool you would run tonight, it has been lapped. Treat it as a research artifact with an MIT license, not a fourth option.

The four stacks side by side

	ik_llama.cpp / llama.cpp	KTransformers v0.7.0	vLLM CPU backend	PowerInfer
License	MIT	Apache 2.0	Apache 2.0	MIT
Runs with zero GPU	✅	❌ GPU required	✅	Partially (GPU-centric design)
Quant formats	GGUF (deep ecosystem)	INT4/INT8 (AMX), FP8	AWQ, GPTQ, INT8 W8A8	Custom sparse formats
Sweet spot	1 user, MoE ≤ ~30B active, any x86/ARM	100B-700B MoE on GPU + big RAM	Batch/concurrent serving on AVX-512 servers	Research
Setup friction	Low (Ollama) to medium (fork build)	High	High	High
Maintenance (Sep 2026)	Very active	Very active	Active (core project)	Last push May 2026

Which stack wins at each memory tier?

Desktop, 64-96GB DDR5 (Ryzen 9 9950X class): ik_llama.cpp, no contest. Dual-channel DDR5 bandwidth caps you around 90 GB/s, which means MoE models with small active parameter counts (Gemma 4 26B class, Qwen3 MoE mid-sizes) at 4-8 tok/s. vLLM CPU adds nothing for one user; KTransformers wants more RAM and a GPU anyway. Boundary: dense 70B models stay unusable at this tier — roughly 1 tok/s territory — whatever software you pick.

EPYC/Threadripper server, 256GB+: this is where the real decision sits. Twelve-channel EPYC moves 400+ GB/s, which turns 400B-class MoE models into single-digit-tok/s realities. CPU-only via ik_llama.cpp works; adding one used 24GB GPU and switching to KTransformers roughly doubles-to-quadruples responsiveness on supported models because attention and KV cache live on the GPU. If the board already exists, the GPU is the cheapest upgrade in the whole system now — RAM is the expensive part.

No hardware yet: do not buy a CPU-inference rig in late 2026 at current DRAM prices to save money — run the numbers first, and test the workload on rented hardware before committing (details in the table below). A used RTX 3090 at ~$1,252 (September 2026) delivers 930 GB/s — ten times desktop DDR5 bandwidth — and beats any new-build CPU box under ~$2,000 total on $/token.

When should you NOT run CPU inference?

• Coding assistants and agent loops. Agents chain dozens of calls; 5 tok/s per step compounds into minutes of waiting. These workloads need 30+ tok/s, which means VRAM.
• Anything with concurrent users on desktop hardware. Bandwidth is shared; two simultaneous chats halve an already-slow number. (A fat EPYC running vLLM CPU is the exception, by design.)
• Dense models above ~13B. The MoE sparsity trick is the entire story. A dense 70B reads every weight every token and no 2026 software changes that.
• If you would have to buy the RAM today. 64GB of DDR5 at $680-$1,070 against a rented 3090 at pennies per hour: the break-even is thousands of hours out. Wait out the DRAM market or rent.

What to actually buy

Prices as of September 2026, all verified in the comparison above:

Your situation	The move	Price	Where
Own a 64GB+ desktop already	ik_llama.cpp + a Q4 MoE GGUF, $0 spend	free	ik_llama.cpp review
Own an EPYC/256GB board, want frontier MoE	Add a used RTX 3090 24GB, run KTransformers	~$1,252	Check price
Desktop at 32GB, tempted to upgrade RAM	64GB DDR5 kit — only if a specific model needs it	$680-$1,070	Check price
Undecided — want to test tok/s before any purchase	Rent a 3090 by the hour and benchmark your model	from $0.07/hr	Vast.ai

For the deeper dollars-per-token hardware analysis, see the CPU vs GPU home-lab breakdown at runaihome.com, and if the end goal is a local coding backend, aicoderscope.com covers wiring local endpoints into Cline and Cursor.

FAQ

Can KTransformers run without any GPU? No. KTransformers v0.7.0 is a CPU-GPU heterogeneous framework; every documented configuration pairs CPU expert offloading with at least one GPU (24GB-class minimum) holding attention layers and KV cache. For zero-GPU inference, use llama.cpp or ik_llama.cpp instead.

Does vLLM’s CPU backend read GGUF files like Ollama does? No. On x86 CPUs, vLLM loads HuggingFace-format weights in FP32/FP16/BF16 and supports AWQ, GPTQ, and INT8 (W8A8) quantization. The GGUF ecosystem belongs to llama.cpp and its derivatives. If a model only exists as a GGUF, vLLM CPU is not an option.

What speed should I expect from CPU-only inference on a normal desktop? Roughly 4-8 tok/s for MoE models with ~3-4B active parameters at Q4 on dual-channel DDR5 (about 80-100 GB/s of bandwidth), and ~1 tok/s for dense 70B models — the generation speed tracks memory bandwidth divided by active weight bytes per token, as of October 2026.

Sources

• KTransformers — github.com/kvcache-ai/ktransformers (v0.7.0, Apache 2.0, checked Oct 2026)
• vLLM CPU installation docs — docs.vllm.ai (x86 backend, data types, AVX-512 guidance)
• AMD PACE vLLM plugin announcement — amd.com
• PowerInfer — github.com/SJTU-IPADS/PowerInfer (MIT, last push May 11, 2026)
• Tom’s Hardware RAM price index (DDR5 kit prices, Sep 2026)

What self-hosting actually costs

Real cost breakdowns for self-hosted AI: hardware floors, power, maintenance hours, and the honest comparison against paying for it. No spam, unsubscribe anytime.

Email address Subscribe