Inkling-Small Self-Hosting Guide 2026: Run 276B on One GPU

By AIFoss · 9 min read
inklingllamacppvllmggufselfhosted

TL;DR: Inkling-Small is a 276B-parameter, 12B-active MoE from Thinking Machines Lab under clean Apache 2.0 — no MAU caps, no revenue clauses. Unsloth’s 2-bit GGUF lands at ~88GB, which makes this the first frontier-class model that fits a single 96GB card or a 128GB unified-memory box. The catch: 2-bit quality tax is real, and KV headroom is tight.

What you’ll have running after this guide:

  • Inkling-Small serving locally through llama-server from an Unsloth GGUF, on hardware you can actually own
  • A vLLM deployment path for multi-GPU rigs, using the same day-0 support the 975B Inkling shipped with
  • A clear read on which quant fits which hardware tier — before you download 88GB over your home connection

Honest take: If you already own 96GB+ of VRAM or a 128GB unified-memory machine, run the UD-Q2_K_XL and see for yourself — the license lets you do anything with it. If you’d be buying hardware for this, rent the exact GPU on RunPod for a weekend first.

License check: Apache 2.0, no asterisks

Start here, because in 2026 “open weights” means five different things. Inkling-Small ships under plain Apache 2.0, the same license Thinking Machines used for the 975B Inkling in July. No monthly-active-user threshold, no revenue trigger, no attribution requirement in your UI, no separate agreement for commercial serving.

That puts it in a different category from Kimi K3, whose MIT-derived license adds a revenue clause and a “Kimi K3” display requirement at scale. Neither clause bites a home-labber, but if you’re building anything commercial on top of a local model, Apache 2.0 is the license your lawyer stops asking questions about.

Verify it yourself rather than trusting a blog: the license field on the thinkingmachines/Inkling-Small Hugging Face model card reads Apache 2.0, matching the parent Inkling release.

What you’re actually downloading

Inkling-Small was released July 30, 2026 — two weeks after the 975B flagship. The numbers that matter for self-hosting:

  • 276B total parameters, ~12B active per token. It’s a sparse MoE, so per-token compute behaves like a 12B model while the full expert pool has to sit somewhere in memory. Disk and RAM are your constraints, not FLOPS.
  • Same architecture as the 975B Inkling. This is why tooling support arrived on day 0 instead of the usual 2–6 week llama.cpp lag — the relative attention, short convolution, and shared expert sink components were already merged for the big model in mid-July 2026.
  • Multimodal in: text, image, and audio. Context window is 1M tokens on paper; at home-lab memory budgets you will run a small fraction of that (more below).
  • It beats its big sibling on some benchmarks. Vendor-reported IFBench is 83.4%, ahead of the 975B flagship, and Thinking Machines’ launch post claims parity or better on agentic and tool-use tasks. Treat those as vendor numbers — no independent coding benchmarks existed at weight release — but the direction matches early r/LocalLLaMA testing.

Official checkpoints come in BF16 plus MXFP8 and NVFP4 variants. The NVFP4 checkpoint is ~180GB and targets a single B300 — relevant if you’re renting, irrelevant if you’re running a home rig. For owned hardware, the Unsloth GGUFs are the practical path.

Quant sizes and the hardware that fits them

Unsloth published dynamic GGUFs the day weights landed. Sizes as of early September 2026 (check unsloth/Inkling-Small-GGUF on Hugging Face before downloading — repacks happen):

QuantSizeFits onRealistic context
UD-IQ1_S~74.8GB80GB VRAM, or 96GB unified16–32K
UD-IQ1_M~78.8GB96GB VRAM (comfortable)16–32K
UD-IQ2_M~82.4GB96GB VRAM8–16K
UD-Q2_K_XL~87.9GB96GB VRAM (tight), 128GB unified8–16K
UD-Q4_K_M~163GB192GB+ multi-GPU or CPU RAMdepends on headroom

Four tiers of hardware make sense here:

  1. Single RTX PRO 6000 (96GB). The headline configuration: UD-Q2_K_XL at 87.9GB leaves roughly 8GB for KV cache and activations. That’s genuinely tight — expect 8–16K usable context, not the 1M on the model card. Drop to UD-IQ2_M if you need more room.
  2. 4× RTX 3090 (~96GB total). Same quant, split across cards with --split-mode layer in llama.cpp or tensor parallel in vLLM. Used 3090s remain the cheapest path to this VRAM total, at the cost of power draw and PCIe topology headaches.
  3. 128GB unified memory (Strix Halo boxes like the GMKtec EVO-X2, or a Mac with 128GB). UD-Q2_K_XL fits with real KV headroom. Bandwidth is the ceiling — expect noticeably lower tok/s than the GPU paths, but it’s the quietest and lowest-power option.
  4. CPU server with 256–512GB RAM. Q4_K_M or better fits, and 12B active parameters keeps CPU inference from being a total slideshow — but “usable for batch jobs” is the honest framing, not “interactive.”

The sister-site hardware guide at runaihome.com goes deeper on the single-card-vs-quad-3090 math if you’re speccing a build.

llama.cpp setup

Inkling architecture support merged into llama.cpp in mid-July 2026 for the 975B release, and Inkling-Small shares the architecture. Anything you build or download from late July 2026 onward works.

Download a quant (Hugging Face CLI shown; aria2 works too and resumes better on big files):

pip install -U "huggingface_hub[cli]"
hf download unsloth/Inkling-Small-GGUF \
  --include "UD-Q2_K_XL/*" --local-dir ./inkling-small

Then serve it:

llama-server -m ./inkling-small/UD-Q2_K_XL/Inkling-Small-UD-Q2_K_XL-00001-of-00002.gguf \
  -ngl 99 -c 16384 --port 8080

Expected startup output includes the architecture line — if you see the model load layers and print something like:

llama_model_loader: - kv   0: general.architecture str = inkling
...
llama_new_context_with_model: n_ctx = 16384
main: server is listening on http://127.0.0.1:8080

you’re live, with an OpenAI-compatible endpoint at :8080/v1.

The failure you’ll actually hit: error loading model: unknown model architecture: 'inkling'. This means your llama.cpp build predates the mid-July 2026 architecture merge. Distro packages and old Docker tags are the usual culprits. Fix: git pull && cmake --build build from source, or grab a current release binary — then confirm with llama-server --version that the build date is recent. The same stale-build error hit people during the 975B launch week, so it’s well documented in the llama.cpp issues.

On the multi-GPU tier, add --split-mode layer and let llama.cpp distribute across the 3090s; check nvidia-smi shows all four cards loaded before blaming the model for slow output.

vLLM setup

Thinking Machines and the vLLM team shipped day-0 support for the whole Inkling family — v0.26.0 added the architecture for the 975B launch, and Inkling-Small runs on the same path. If you’re on the 4× GPU tier or renting, vLLM is the better server: continuous batching, proper concurrency, and support for the official MXFP8/NVFP4 checkpoints instead of GGUF.

pip install -U vllm
vllm serve thinkingmachines/Inkling-Small \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.95 \
  --max-model-len 32768

Cap --max-model-len deliberately. vLLM preallocates KV cache, and at 96GB total VRAM with a 2-bit-class footprint you don’t have room for the default long-context allocation — start at 32K and raise it only if startup succeeds with margin. Check the model card for any required --trust-remote-code or parser flags before filing bugs; day-0 support means the architecture works, not that every flag combination is tested.

For quantized vLLM serving on smaller rigs, the tradeoffs are the same ones covered in our GPTQ vs AWQ vs GGUF production guide.

When not to self-host this

Be honest about three things before committing 88GB of disk and a hardware budget:

  • 2-bit is a quality tax. Unsloth’s dynamic quants are the best-in-class at this size, but nobody serious claims 2-bit matches the BF16 checkpoint. For code generation especially, test your real workload before trusting it.
  • Context is the hidden constraint. Every tier above runs a sliver of the advertised 1M window. If your use case is whole-repo RAG or 200K-token documents, the memory math changes completely — see our quantization guide for the KV-size arithmetic.
  • Renting beats buying for evaluation. A weekend on RunPod with an H200 or B300 running the official NVFP4 checkpoint tells you what the model can actually do at full quality — then you’ll know whether the 2-bit local version’s output is the model or the quant. Do that before spending four figures on hardware.

If your ceiling is 24–32GB of VRAM, this model isn’t your move at any quant. The 975B Inkling guide covers the family’s top end; below 96GB, a dense 27–70B model remains the better daily driver.

FAQ

Does Inkling-Small’s Apache 2.0 license allow commercial use? Yes, without conditions. Apache 2.0 permits commercial serving, fine-tuning, and redistribution with no revenue thresholds or attribution-in-UI requirements — unlike Kimi K3’s modified MIT or the various “open” licenses with MAU caps.

Can I run Inkling-Small on a single RTX 4090? No. The smallest Unsloth quant is ~74.8GB; a 24GB card can’t hold it, and CPU offload of a 276B MoE at that ratio yields single-digit tok/s at best. 96GB of VRAM or 128GB of unified memory is the realistic floor.

Is Ollama support available? Not at publish time — Ollama’s model library hadn’t listed Inkling-Small as of early September 2026, and its runner lags llama.cpp architecture merges. Use llama-server directly; it exposes the same OpenAI-compatible API that most Ollama front-ends can point at.

Sources

Was this article helpful?