TL;DR: ik_llama.cpp is an MIT-licensed fork of llama.cpp focused on CPU performance and state-of-the-art quantization, maintained by the developer who wrote most of mainline’s quant types. On CPU it roughly doubles mainline’s token generation in the project’s own comparisons, and it just made news pushing 5 tok/s from Gemma 4 26B on a 13-year-old Xeon. The trade: you build from source, and you give up mainline’s ecosystem breadth.

	ik_llama.cpp	Mainline llama.cpp	Ollama
Best for	CPU-only and RAM-offload rigs, big MoE models	Broadest hardware and model support	Zero-effort model management
Install effort	Build from source (two cmake commands)	Prebuilt releases or source	One installer
Hardware support	CPU (AVX2+/ARM NEON), CUDA (Turing+) only	CPU, CUDA, ROCm, Vulkan, Metal, SYCL	Everything llama.cpp supports
The catch	No Metal/ROCm/Vulkan, smaller ecosystem	Slower CPU decode, fewer quant types	No control over the engine

Honest take: If your inference box is a GPU-rich desktop, stay on mainline or Ollama. If you’re squeezing a big MoE model through system RAM — or running with no GPU at all — ik_llama.cpp is the single biggest free speedup available in 2026.

The r/LocalLLaMA hardware meta in 2026 has quietly split in two. One camp buys VRAM. The other camp noticed that modern MoE models only activate a few billion parameters per token and started asking how far plain DDR memory and a fat CPU can go. ik_llama.cpp is the second camp’s favorite tool, and a June 2026 blog post from Neomind Labs — Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU — put it on the Hacker News front page and back on everyone’s radar.

This review covers what the fork actually is, why it’s faster on CPU, how to build it, and the honest list of reasons to stay on mainline.

What ik_llama.cpp is

ik_llama.cpp is a fork of llama.cpp created in June 2024 by Iwan Kawrakow — the “ik” in the name, and the developer behind most of mainline llama.cpp’s quantization formats (the K-quants carry his initials). The repo describes itself as a “llama.cpp fork with additional SOTA quants and improved performance,” and it has a track record of shipping inference features before mainline does. It’s MIT-licensed, sitting at roughly 3.1k stars and 420+ forks as of August 2026 — small next to mainline’s six figures, but active and focused.

Three things separate it from the project it forked from:

More aggressive quantization types. Beyond mainline’s GGUF quants, ik_llama.cpp ships the IQK series (IQ2_K through IQ6_K), Trellis quants (IQ1_KT through IQ4_KT), and MXFP4. These target better quality-per-bit than mainline equivalents at the same size — the same design philosophy Kawrakow brought to K-quants, iterated further. If the difference between a model fitting in your RAM or not is 8%, these quants are the 8%.

CPU-first attention optimizations. The fork implements Flash Attention on CPU, plus MLA (multi-head latent attention) with FlashMLA variants that the project bills as the fastest CPU-only path for DeepSeek-family models. Prompt processing — historically the most painful part of CPU inference — is substantially faster across quant types, and there’s run-time repacking that reorders weights into CPU-friendly layouts at load.

Current model support despite the fork date. The codebase last fully synced with upstream in August 2024, but model support has been maintained independently and is genuinely current: DeepSeek V3/V4, the Qwen3 series, GLM-4.x and 5.x, Gemma, MiniMax-M3, and dozens more. Features like multi-token-prediction decoding, speculative decoding, and auto-fit VRAM for MoE offloading are all in.

The numbers that matter

The project’s own comparison threads (discussion #164) show quantized token generation at roughly 5.05 tok/s versus mainline’s 2.70 tok/s on a Xeon E5-2683 v4 — close to a 2x gap on the same hardware. Treat that as project-reported, but the independent data points agree on direction.

The Neomind Labs result is the one worth understanding, because it explains why this works. Ryan Findley ran Gemma 4 26B — a mixture-of-experts model that activates only ~4B parameters per token — on a 13-year-old Xeon server with 128GB of DDR3, total hardware cost under $300, and measured 5.2 tok/s decode. That number isn’t magic; it’s memory-bandwidth math. DDR3 in that class of machine moves roughly 40–50 GB/s, each generated token requires reading the active weights once, and 5 tok/s is about what the arithmetic predicts. CPU inference in 2026 is bandwidth-bound, not compute-bound, and ik_llama.cpp’s contribution is keeping the CPU efficient enough that you actually hit the bandwidth ceiling instead of stalling below it.

One caveat from that experiment: the fork’s headline optimizations assume AVX2 and FMA3, which Intel introduced with Haswell in 2014. The Neomind Xeon predates that, so the --run-time-repack flag had to be dropped — it works anyway, but pre-AVX2 silicon is edge-case territory. The README’s official floor is “AVX2 or better” on x86 and ARM NEON on ARM.

Building it

There are no one-line installers here. You clone and build — which on Linux is two commands past the clone:

git clone https://github.com/ikawrakow/ik_llama.cpp
cd ik_llama.cpp
cmake -B build -DGGML_NATIVE=ON
cmake --build build --config Release -j$(nproc)

-DGGML_NATIVE=ON matters: it compiles for your exact CPU’s instruction set, which is where a chunk of the speedup lives. A binary built without it (or built on a different machine) leaves performance on the table.

For a hybrid rig with an NVIDIA card, add CUDA:

cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)

CUDA support requires a Turing-generation GPU or newer. The resulting binaries mirror mainline’s — llama-server gives you the OpenAI-compatible HTTP endpoint, so anything you already point at llama.cpp or Ollama (Open WebUI, Continue.dev, LibreChat) connects the same way, typically on port 8080 with ./build/bin/llama-server -m model.gguf.

Standard GGUF files work as-is. To get the fork-exclusive IQK and Trellis quants you either download them where available (several Hugging Face quantizers publish ik-specific formats for big MoE releases) or requantize yourself with the included tools. Our GGUF quantization guide covers the base formats if the alphabet soup is new.

When to use it — and when not to

Use ik_llama.cpp when:

• You have no GPU, or not enough. This is the headline case. A retired dual-socket Xeon workstation or any modern desktop with 64–128GB of RAM becomes a usable MoE-model host. On current hardware it’s even better: a Ryzen 9 7950X with dual-channel DDR5 roughly doubles the old Xeon’s bandwidth, and AVX-512 support lets the fork’s kernels stretch further.
• You’re offloading a big MoE to system RAM. Running DeepSeek V4 Flash or GLM-5.2 with experts in RAM and shared layers on a 24GB card is exactly the workload the FlashMLA and repacking work targets. Our DeepSeek V4 self-hosting guide covers that offload pattern on mainline; the same setup on ik_llama.cpp is the known upgrade path.
• You want maximum quality per gigabyte. The IQK and Trellis quants let a model tier fit where mainline’s equivalent quant wouldn’t.

Stay on mainline (or Ollama) when:

• You’re on Apple Silicon, AMD, or Intel Arc. There is no active Metal, ROCm, or Vulkan support in the fork. A Mac should run mainline or an MLX stack; an AMD GPU should run mainline’s ROCm/Vulkan builds.
• Your GPU already fits the model. If the whole model sits in VRAM on a RTX 3090 or better, CUDA decode speed is similar either way and mainline’s ecosystem (day-one model conversions, prebuilt binaries, distro packages) wins on convenience. For serious GPU throughput, vLLM on a rented card is the ceiling anyway — a RunPod A100 costs a few dollars an hour with nothing to build.
• You want model management, not an engine. ik_llama.cpp is a compiler target, not a product. No model registry, no pull command, no auto-updates. That’s Ollama’s job — see our runner comparison for that trade-off.

The structural risk is the usual one-maintainer-fork risk: divergence. The last full upstream sync was August 2024, so mainline features land here only when ported deliberately, and brand-new architectures occasionally arrive later than mainline (sometimes earlier — it cuts both ways). For a production service, that’s a real dependency question. For a home lab, it’s a non-issue: the engine you built today keeps working.

If you’re speccing hardware around this — memory channels matter more than CPU cores for this workload — our sister site runaihome.com covers the home-lab hardware side, including what Gemma 4 needs on CPU-only builds.

Verdict

ik_llama.cpp does one thing and does it better than anything else: extract the maximum from CPUs and system RAM. The 2026 shift toward sparse MoE models made that skill dramatically more valuable — a machine that could never run a 26B dense model at usable speed handles a 26B-A4B MoE fine, and this fork is the difference between 2.7 and 5 tok/s on the same silicon. MIT license, clean build, OpenAI-compatible server, honest engineering culture. If your inference budget is RAM instead of VRAM, this belongs in your stack. Pair it with the QAT checkpoints from our Gemma 4 QAT guide and a $0 software stack runs a current-generation model on hardware most offices throw away.

FAQ

Is ik_llama.cpp a drop-in replacement for llama.cpp? Mostly. It builds the same binaries (llama-server, llama-cli), serves the same OpenAI-compatible API, and reads standard GGUF files. What changes is hardware coverage — CPU and CUDA only — and the build-from-source requirement. Client apps pointed at the server won’t notice the swap.

Do I need special model files to benefit? No. Standard GGUF quants already run faster, especially prompt processing and MoE offload. The fork-exclusive IQK and Trellis quants are an additional gain on top, worth using when you’re tight on memory.

How much RAM do I need for CPU-only inference? Enough to hold the quantized model plus context — and then bandwidth decides your speed. A 26B-A4B MoE at Q4 fits comfortably in 32GB; big MoEs like DeepSeek V4 Flash want 128–256GB. Rough decode estimate: memory bandwidth in GB/s divided by active-parameter bytes per token.

Recommended Gear

• Ryzen 9 7950X — 16 cores with AVX-512 and dual-channel DDR5; a strong modern CPU-inference platform.
• RTX 3090 — the used-market 24GB baseline if you’d rather add VRAM than tune CPU inference.

Sources

• ik_llama.cpp GitHub repository — README, license, build instructions, feature list
• ik_llama.cpp discussion #164: CPU performance comparison with llama.cpp
• Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU — Neomind Labs
• Hacker News discussion of the Neomind Labs post

What self-hosting actually costs

Real cost breakdowns for self-hosted AI: hardware floors, power, maintenance hours, and the honest comparison against paying for it. No spam, unsubscribe anytime.

Email address Subscribe