TL;DR: vLLM’s tensor parallelism turns two 24GB cards into one 48GB inference server, and on current builds (tested against v0.26.0, released July 25, 2026) the setup is finally reasonable for home labs. The trade-off is configuration complexity — three flags decide whether you get a stable server or an OOM loop. Worth it if you serve concurrent requests; skip it if you’re the only user.

	vLLM tensor parallel (TP=2)	Two separate Ollama instances	vLLM pipeline parallel
Best for	One large model (30B–70B quantized), concurrent users	Two smaller models, single user, zero fuss	Mismatched GPUs (e.g., 3090 + 4060 Ti)
Setup effort	Medium — flags matter, NCCL can bite	Trivial	Medium, slower than TP
The catch	Both GPUs must be identical class; VRAM headroom math is on you	Can’t run a model bigger than one card	Higher latency; only worth it when TP isn’t possible

Honest take: If you built a dual-GPU box to run 30B+ models with real concurrency, vLLM tensor parallel is the point of that hardware. If you just want two models loaded at once, keep running two Ollama instances and save yourself an afternoon.

A note on versions before anything else: several guides circulating on Reddit and dev.to this summer reference “vLLM v0.6” scheduler improvements. vLLM 0.6 shipped in September 2024 — two years ago. As of this writing the current release is v0.26.0 (July 25, 2026), preceded by v0.25.0 on July 11. Everything below was checked against the current engine arguments; the scheduler and memory flags here exist in all recent releases.

What tensor parallelism actually does

With --tensor-parallel-size 2, vLLM splits every weight matrix across both GPUs. Each layer’s computation happens on both cards simultaneously, with results synchronized over PCIe (or NVLink if you have it). The practical effect: a model that needs ~40GB of weights and KV cache runs on two 24GB cards, and both GPUs contribute compute to every single token.

This is different from what Ollama does when it splits a model across GPUs — Ollama’s llama.cpp backend assigns whole layers to each card, so GPUs take turns rather than working together. Tensor parallel keeps both cards busy at once, which is why vLLM’s throughput advantage shows up hardest under concurrent load. One widely shared June 2026 benchmark report measured a 19× throughput gap between vLLM and Ollama on identical hardware under batch load; treat the exact number with suspicion (batching setups vary wildly), but the direction is real and repeatable.

The reference build for this guide is two RTX 3090 cards — still the price-per-VRAM king in 2026, and the last consumer GeForce card with NVLink support. A dual RTX 4090 build works identically (no NVLink, but PCIe 4.0 x8/x8 is fine for TP=2). For the full hardware side — PSU sizing, slot spacing, thermals — see the multi-GPU build guide on runaihome.com.

Step 1: Install and launch with TP=2

Fresh virtual environment, then:

pip install vllm

Launch a 32B model quantized to AWQ, split across both cards:

vllm serve Qwen/Qwen3-32B-AWQ \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 16384 \
  --max-num-seqs 16

For single-node dual-GPU, that’s all the parallelism config you need — vLLM handles worker orchestration itself and no Ray cluster setup is required. If startup hangs at NCCL initialization (common on consumer boards where PCIe peer-to-peer is flaky), relaunch with NCCL_P2P_DISABLE=1 in the environment. You lose a little synchronization speed and gain a server that actually starts.

Step 2: Verify both GPUs are actually working

Don’t trust the startup logs — check utilization during inference. Send a request, then in another terminal:

watch -n 1 nvidia-smi

You want to see both cards at roughly equal memory consumption and both showing compute utilization while a request is in flight. If GPU 1 sits at 0% utilization with memory allocated, your requests are running single-GPU and something in the config regressed. Equal memory but idle compute on one card usually points at the NCCL issue above.

Step 3: The three flags that prevent OOM crashes

Multi-GPU vLLM crashes are almost never mysterious — they come from one of three defaults being wrong for 24GB cards.

--gpu-memory-utilization — vLLM pre-allocates this fraction of each GPU for weights plus KV cache. The default 0.9 works on datacenter cards with nothing else running; on a desktop where your display server holds 500MB–1GB on GPU 0, it’s a crash waiting for boot. Start at 0.85. If you run headless, push to 0.92 and stop when startup fails.

--max-model-len — context length directly sizes the KV cache reservation. Model cards advertising 128K context will happily try to reserve KV cache for it and fail. 16384 is a sane ceiling for a dual-24GB box running a 32B model; drop to 8192 if you want more concurrent sequences instead of longer ones.

--max-num-seqs — caps concurrent sequences in a batch. The default (256 in recent releases) is a datacenter number. At 16, memory spikes under load stay bounded and per-request latency stays predictable on consumer cards.

The interaction between these three is the whole game: total VRAM budget = weights + (KV cache per token × max-model-len × max-num-seqs, roughly). If you tighten one you can loosen another. Our GPTQ vs AWQ vs GGUF comparison covers picking a quant that leaves enough KV cache headroom in the first place.

Step 4: Scheduler configuration

vLLM’s scheduler decides which queued requests join the running batch each step. Two policies exist, set via --scheduling-policy:

• fcfs (default) — first come, first served. Correct for a single user or a handful of interactive users.
• priority — requests carry a priority value (lower runs earlier), with arrival time breaking ties. Useful when your box serves mixed traffic: an interactive chat UI plus a background batch job (nightly embedding runs, log summarization) that shouldn’t starve the humans.

Client-side, priority is passed per-request through the API. If everything hitting your server is the same kind of traffic, leave fcfs alone — priority scheduling only earns its keep when workloads genuinely compete. What you should not expect from either policy is an OOM fix; the scheduler works within the memory budget set by the three flags above, and no scheduling policy rescues an over-committed KV cache.

When two Ollama instances are still the right answer

Dual-GPU vLLM is the wrong tool in three common situations:

1. You’re the only user. Ollama with layer-split across both cards gives you a 70B Q4 at interactive speeds with zero configuration. Batching throughput — vLLM’s core strength — buys you nothing at concurrency of one. Our Ollama vs vLLM comparison covers this in depth.
2. You want two different models resident. One Ollama instance per GPU (set CUDA_VISIBLE_DEVICES per instance) is simpler and doesn’t tie the cards together.
3. Your GPUs are mismatched. Tensor parallel wants identical cards; splitting a matrix between a 24GB and a 16GB card wastes the difference. Pipeline parallelism (--pipeline-parallel-size 2) tolerates asymmetry but adds latency — it’s the fallback, not the goal.

If your workload only needs big-GPU power occasionally, renting beats building: an A100 80GB on RunPod runs a 70B without any parallelism gymnastics, and you pay only for hours used. For always-on serving with authentication and monitoring on top of this setup, continue with the vLLM production setup guide.

FAQ

Do I need NVLink for tensor parallel on two RTX 3090s? No. TP=2 synchronization traffic fits comfortably in PCIe 4.0 x8/x8 bandwidth for inference. NVLink helps most for training and for TP across 4+ cards. If you already own the bridge, use it; don’t buy one just for this.

Can I mix an RTX 3090 with an RTX 4090? Tensor parallel will run but allocates symmetrically, so the 4090 behaves like a second 3090 and its extra speed is mostly wasted waiting at sync points. Mismatched pairs are better served by pipeline parallelism or by running separate instances per card.

Why does vLLM crash on startup when Ollama runs fine on the same machine? vLLM pre-allocates its memory budget up front instead of growing on demand, so misconfiguration fails at boot rather than mid-request. Check that Ollama (or anything else) isn’t holding VRAM when vLLM starts, and lower --gpu-memory-utilization to 0.85.

Sources

• vLLM releases on GitHub — v0.26.0, July 25, 2026
• vLLM engine arguments documentation — --scheduling-policy, --gpu-memory-utilization, --max-num-seqs
• vLLM priority scheduling RFC (#6077)
• Setting up vLLM with dual RTX 3090s — independent walkthrough of the same hardware class

Recommended Gear

• RTX 3090 — 24GB VRAM, NVLink-capable, the used-market value pick for dual-GPU inference builds
• RTX 4090 — faster per card if you’re buying new; pair two for TP=2 over PCIe