TL;DR: AirLLM (Apache 2.0, ~37.7k GitHub stars as of October 2026) really does run Llama 3.3 70B on a 4GB GPU — by keeping only one transformer layer in VRAM at a time and streaming the rest from disk. The price is that every generated token re-reads the entire model from your SSD, so community-reported speeds range from roughly 0.7 tokens/second on a fast NVMe desktop down to 0.07 tok/s on weaker hardware. That makes AirLLM a batch and testing tool, not a chat replacement. If you have 64GB of RAM, llama.cpp CPU offload is several times faster for the same model; if you want 70B interactively, no 4GB trick gets you there.

What is AirLLM and how does it run 70B models in 4GB of VRAM?

AirLLM (github.com/lyogavin/airllm) is an Apache 2.0 Python library that performs layer-wise inference: it first splits a downloaded Hugging Face model into per-layer shards on disk, then during generation loads layer 0 into the GPU, computes it, frees it, loads layer 1, and so on. Peak VRAM is one layer’s weights plus activations — about 4GB for a 70B model at full precision, per the project README. No quantization, distillation, or pruning is required, which is the headline: you get the full-precision model’s output quality on a GPU that could never hold it.

The README’s claims scale further than 70B: Llama 3.1 405B in ~8GB, Qwen3-235B in ~3GB, DeepSeek-V3 671B in ~12GB. Supported families as of October 2026 include Llama 2/3/3.1/3.3/4, Qwen through 3.8 (including MoE and FP8 variants), DeepSeek V2/V3/R1, Mistral, Mixtral, Gemma, Phi, and InternLM. The project added FP8 support in v3.0 (June 2026) and training support in September 2026. It installs with pip install airllm, runs entirely locally after the model download, and has no hosted tier.

The catch is not whether it works. It is how long each token takes.

How slow is AirLLM really?

Expect seconds per token, not tokens per second. Decode speed in any LLM runtime is bounded by how fast weights reach the compute unit, and AirLLM’s weights live on disk:

tok/s ceiling = storage read speed (GB/s) ÷ weights read per token (GB)

A 70B model is ~140GB at FP16, ~35GB with AirLLM’s optional 4-bit block-wise compression. On a Gen4 NVMe drive reading ~7GB/s, the arithmetic gives a hard floor of ~20 seconds per token at FP16 and ~5 seconds per token at 4-bit — before any compute or Python overhead. On a SATA SSD (~0.55GB/s), FP16 is over four minutes per token. A hard drive is not usable at all.

Community-reported numbers match that math. Third-party write-ups (see Sources; no controlled benchmark exists from the project itself) report roughly 0.7 tok/s as the best case for a quantized 70B on an NVMe desktop, ~0.07 tok/s on a 16GB M2 MacBook Pro, and ~5.3 seconds per token in 50-prompt batch mode. Treat all of these as rough — methodology varies and your disk matters more than your GPU.

Setup (Llama 3.3 70B class)	Realistic decode speed	Source of limit
AirLLM, 4-bit, 4GB GPU + Gen4 NVMe	~0.1–0.7 tok/s	disk read (~7GB/s)
AirLLM, FP16, 4GB GPU + Gen4 NVMe	~0.05 tok/s	disk read
llama.cpp Q4_K_M in 64GB DDR5 RAM (CPU)	~1–2 tok/s (ceiling 2.1)	RAM bandwidth (~90GB/s)
RTX 3090/4090 24GB + CPU offload	2–5 tok/s	PCIe + RAM
RTX PRO 6000 96GB (fits fully)	35–40 tok/s	VRAM bandwidth

Interactive chat needs roughly 20+ tok/s to feel usable. Nothing on a 4GB card gets within two orders of magnitude of that for 70B. That boundary is the honest frame for everything below.

How do you set up AirLLM on Linux?

Install the package, then point it at a Hugging Face model ID. Llama weights are gated, so you need a HF token with license acceptance:

pip install airllm bitsandbytes

from airllm import AutoModel

model = AutoModel.from_pretrained(
    "meta-llama/Llama-3.3-70B-Instruct",
    compression='4bit',          # 3x+ faster than FP16; needs bitsandbytes
    hf_token='hf_...',
    layer_shards_saving_path='/mnt/nvme/airllm-shards'
)

input_tokens = model.tokenizer(
    ["What is the capital of Australia?"],
    return_tensors="pt", truncation=True, max_length=128
)
out = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=20, use_cache=True, return_dict_in_generate=True
)
print(model.tokenizer.decode(out.sequences[0]))

On first run AirLLM downloads the full model (~140GB for 70B FP16) and then writes the per-layer split — plan for ~280GB+ free disk during that step, on the NVMe drive, since the shard path is also what gets streamed at inference time. The layer_shards_saving_path option exists precisely so the split does not land in your root-partition Hugging Face cache; delete_original reclaims the download afterward.

A problem you will likely hit: generation appears to hang after the model loads. Check nvidia-smi — if VRAM is at ~3–4GB and fluctuating, it is not hung, it is generating at seconds per token. Set profiling_mode=True to get per-layer timing output and confirm disk read is the bottleneck. Also note that prefetching (overlapping disk reads with compute) is on by default but, per the README, only the AirLLMLlama2 class supports it — other model families run strictly sequential.

Is AirLLM faster than llama.cpp CPU offload?

No — if the quantized model fits in your RAM, llama.cpp is roughly 3–10x faster, because RAM is faster than disk. Both approaches stream weights the GPU can’t hold; the difference is where from. Dual-channel DDR5 moves ~90GB/s against an NVMe drive’s ~7GB/s. For a 42.5GB Q4_K_M 70B, llama.cpp on a 64GB-RAM machine lands at 1–2 tok/s, versus AirLLM’s ~0.2 tok/s from the same class of hardware. We covered the RAM-and-hybrid path in detail in CPU LLM inference in 2026.

What changed in AirLLM’s favor is the DRAM price crisis. As of October 2026, a 64GB DDR5 kit runs $680–$1,070 (verified September 2026) — RAM stopped being the cheap part of this equation. If your machine has 16GB of RAM, a 4GB GPU, and a fast NVMe drive, AirLLM is the only way to run a 70B at all without buying anything, and “free but overnight” is a legitimate tier. Also note AirLLM runs the unquantized model if you let it: for workloads where quantization quality loss matters, that is a real differentiator — llama.cpp at FP16 would need ~140GB of RAM.

When is AirLLM actually useful — and when should you skip it?

AirLLM fits three use cases: overnight batch jobs (summarize 200 documents, generate a dataset, score outputs), checking a big model’s output quality before committing to hardware, and squeezing inference out of an old 4GB card (GTX 1650, RTX 3050) that cannot do anything else. It is the wrong tool for chat, coding assistants, agents, or anything a person waits on — see when not to self-host AI for the broader decision.

Prices below verified October 2026:

Your situation	Do this	Cost
Old 4GB GPU, fast NVMe, batch workloads overnight	AirLLM, 4-bit compression	$0
Want a usable daily local LLM on a budget	Used RTX 3060 12GB + an 8–13B model in Ollama	~$260–$296 — check price
Need 70B-class output interactively, sometimes	Rent a 24GB GPU, from ~$0.07/hr	Vast.ai, pay per hour

The middle row is the one most 4GB-card owners actually want: a used RTX 3060 12GB runs Qwen-class 8–13B models at fully interactive speeds, and for everyday Q&A those models now beat waiting 90 seconds per 70B sentence. For unified-memory boxes that fit large models in RAM outright, see the Ryzen AI Max 395 guide on runaihome.com.

Boundary cases where AirLLM is not the answer even for batch: MoE models you could fit quantized (a 4-bit Qwen3.6-35B-A3B fits in 24GB and decodes fast), Macs with enough unified memory (load it normally), and any machine whose “SSD” is SATA — the math above makes that a minutes-per-token experience.

FAQ

Does AirLLM degrade output quality? No. Without compression it runs the original full-precision weights — identical outputs to a datacenter GPU, just slower. The optional 4-bit/8-bit block-wise compression trades some quality for a ~3x speedup, similar in spirit to GGUF quantization.

Does AirLLM phone home or require an account? The README documents no telemetry or hosted service; it is a pip package (Apache 2.0) that pulls weights from Hugging Face. After the model download it runs offline. We have not audited the source line by line — air-gapped users should.

How much disk space does a 70B model need with AirLLM? About 140GB for the FP16 download plus a similar amount for the layer-split copy during conversion, so budget ~280GB free NVMe space. Use delete_original=True to drop back to ~140GB afterward.

Sources

• AirLLM GitHub repository — license, supported models, README claims (accessed Oct 13 2026)
• AirLLM runs a 70B model on a 4GB GPU — dev.to/arshtechpro
• Run 70B LLM on 4GB GPU: AirLLM’s real tradeoff — dev.to/umesh_malik
• Layer-by-layer inference with AirLLM (2026) — tutorials.technology

What self-hosting actually costs

Real cost breakdowns for self-hosted AI: hardware floors, power, maintenance hours, and the honest comparison against paying for it. No spam, unsubscribe anytime.

Email address Subscribe