NVIDIA Nemotron 3 Ultra Self-Hosting Guide 2026: What Running the 550B Model Actually Takes

By AIFoss · 8 min read
nemotronnvidiaollamavllmselfhostedgpu

TL;DR: Nemotron 3 Ultra (550B total, 55B active) is genuinely open — weights, training data, and recipes ship under the permissive OpenMDW-1.1 license — but “open” does not mean “runnable at home.” Ollama only exposes it as a cloud passthrough, and the NVFP4 checkpoint needs Blackwell-generation hardware. Your realistic options: the cloud tag, rented multi-GPU nodes, or the smaller Nemotron siblings that actually fit on one card.

What you’ll have running after this guide:

  • Nemotron 3 Ultra answering prompts through Ollama’s nemotron-3-ultra:cloud tag (no weights on your disk)
  • A clear-eyed picture of what true self-hosting costs — which GPUs support NVFP4, roughly how much VRAM the 550B checkpoint needs, and the vLLM route for rented nodes
  • Nemotron 3 Nano running fully locally on a single 24GB GPU as the honest fallback

Honest take: Unless you have a Blackwell server budget, treat Nemotron 3 Ultra as a cloud model and run Nemotron 3 Nano locally. The Ultra weights being downloadable doesn’t make them deployable on anything you own.

What Nemotron 3 Ultra actually is

NVIDIA released Nemotron 3 Ultra on June 4, 2026. It’s a 550B-parameter Mixture-of-Experts model with 55B active parameters per token, built on a hybrid Mamba-Transformer architecture: interleaved Mamba-2 layers, latent MoE blocks, and a small number of attention layers. That hybrid design is why NVIDIA claims dramatically higher inference throughput than dense-attention competitors at long context — its own numbers show 5.9x the throughput of GLM-5.1-754B and 1.6x that of Qwen-3.5-397B on an 8k-input/64k-output workload. Vendor benchmarks deserve skepticism, but the architectural argument is sound: Mamba layers don’t pay the quadratic attention tax, which matters enormously at this model’s headline feature — a 1M-token context window.

The release includes four checkpoints: an NVFP4 quantized version (the one NVIDIA wants you to serve), BF16 instruct, BF16 base, and a generative reward model. Training data and post-training recipes ship alongside the weights, which is more than most “open” releases can say.

It’s positioned squarely at long-running agentic work — multi-hour coding sessions, tool-use chains, and workflows where a model has to hold hundreds of steps of state. If you’ve read our context window guide, you know 1M tokens is far past what most local setups can even allocate KV cache for. That’s foreshadowing.

The license: OpenMDW-1.1, not Apache 2.0

Several early write-ups (and more than a few Reddit threads) called this an Apache 2.0 release. It isn’t. Nemotron 3 Ultra ships under OpenMDW-1.1, the Linux Foundation’s Open Model, Data & Weights license.

The practical read: OpenMDW-1.1 is a permissive license in the Apache 2.0 spirit — commercial use, modification, and redistribution are allowed, and it covers the weights, the training data, and the training recipes as a package rather than weights alone. For self-hosters and businesses, that’s a meaningfully cleaner grant than NVIDIA’s older Open Model License, which carried more strings. If your legal team needs to sign off, point them at the OpenMDW text itself rather than secondhand “it’s basically Apache” summaries — including this one.

Verdict on the license question: genuinely open, fine for commercial self-hosting, just cite the right license name in your compliance docs.

The Ollama situation: cloud passthrough, not a download

Here’s where the popular framing falls apart. Ollama announced Nemotron 3 Ultra support on its blog, and you can run it today:

ollama run nemotron-3-ultra:cloud

But note the tag. There is no local download tag for Nemotron 3 Ultra in the Ollama library — only :cloud. Pulling it fetches a manifest of a few kilobytes; the 550B weights stay on Ollama’s and NVIDIA’s infrastructure, and your prompts travel over the network. It behaves like a local model in your terminal and in Ollama’s API, which is convenient for wiring into existing tooling, but let’s be precise about what it is: a hosted API with an Ollama-shaped interface.

If your reason for self-hosting is privacy or data sovereignty, the :cloud tag does not deliver that. Your prompts leave your machine. For casual evaluation it’s the fastest path — zero setup, works on a laptop — but it belongs in the “cloud API” column of your mental spreadsheet, not the “self-hosted” one.

Can your consumer GPU handle it? Let’s do the math

The NVFP4 checkpoint stores weights at roughly 4 bits per parameter. For 550B parameters that’s on the order of 280GB of weights before you allocate a single byte of KV cache — and a 1M-token context wants a lot of KV cache, even with Mamba layers easing the burden. The BF16 checkpoints are around four times larger.

No consumer GPU comes close. A RTX 5090 has 32GB of VRAM; you’d need roughly nine of them just for weights, and consumer boards don’t have the interconnect to make that sane. This is a model sized for B200/B300-class data center nodes.

There’s a second gate besides capacity: NVFP4 is a Blackwell-only format. Hardware-accelerated NVFP4 requires fifth-generation Tensor Cores — B200, B300, RTX PRO 6000 Blackwell, and the RTX 50 series. A RTX 4090 (Ada generation) tops out at FP8 acceleration, and anything older falls back further. The queue of “will it run on my 4090” questions has a one-word answer, and it isn’t the fun one. If you’re weighing a GPU upgrade for local AI generally, our sister site runaihome.com covers the hardware side in depth.

GPUVRAMNative NVFP4?Nemotron 3 Ultra locally?
RTX 309024GBNoNo
RTX 409024GBNo (FP8 max)No
RTX 509032GBYesNo — capacity, not format
RTX PRO 6000 Blackwell96GBYesOnly in multi-GPU nodes
B200 (data center)192GBYesYes, 2+ GPUs

Actually self-hosting it: the rented-node route

If you have a real reason to run the weights yourself — fine-tuning, air-gapped inference, compliance — the working recipe is a multi-GPU cloud node, not home hardware. The checkpoints on Hugging Face serve through vLLM, SGLang, and TensorRT-LLM; our vLLM setup guide covers the server basics, and the same OpenAI-compatible endpoint pattern applies here with tensor parallelism across GPUs. Check NVIDIA’s Nemotron usage cookbook on GitHub for the current serving flags — they change fast enough that anything I print here would go stale.

Renting is the only sane way to trial this: a two- or four-GPU B200 node on RunPod lets you validate the model against your actual workload for tens of dollars instead of committing to six-figure hardware. If the model earns its keep, you’ll know before you buy anything. If NVFP4 nodes aren’t available in your region, the BF16 checkpoint on H100s works — you’ll just pay the FP8-era memory bill.

The realistic local path: Nemotron 3 Nano and Super

The Ultra release overshadowed the practical news: the smaller Nemotron 3 models have real local download tags on Ollama.

ollama run nemotron-3-nano

Nano is roughly a 24GB pull that fits a single RTX 3090-class card and keeps the 1M-token context window and the agentic post-training that make the family interesting. nemotron-3-super sits between Nano and Ultra for multi-GPU home rigs. If you’re going to quantize further or pick tags deliberately, our GGUF quantization guide applies directly.

For most readers of this site, Nano-locally-plus-Ultra-via-cloud is the setup that survives contact with reality: private inference for everyday work, escalation to the 550B model when a task genuinely needs frontier reasoning.

When not to bother

Skip Nemotron 3 Ultra entirely if: your workloads fit in 128k context (cheaper models do fine); you need strict data locality but can’t rent compliant infrastructure (the :cloud tag violates the requirement by design); or you were hoping for a single-GPU flagship — that niche belongs to Nano and Super. And if you just want the strongest thing your 24GB card can run today, this release changes nothing for you.

FAQ

Is Nemotron 3 Ultra really open source? The weights, training data, and recipes are released under OpenMDW-1.1, a permissive Linux Foundation license that allows commercial use and modification. It’s about as open as large model releases get in 2026 — just not Apache 2.0, despite what early coverage claimed.

Can I download Nemotron 3 Ultra through Ollama? No. Ollama offers only the nemotron-3-ultra:cloud passthrough tag, which runs the model on remote infrastructure. To run the actual weights you need the Hugging Face checkpoints plus vLLM, SGLang, or TensorRT-LLM on multi-GPU hardware.

Does NVFP4 work on an RTX 4090? No. Hardware-accelerated NVFP4 requires Blackwell’s fifth-generation Tensor Cores (RTX 50 series, RTX PRO 6000, B200/B300). Ada-generation cards like the 4090 support FP8 at best — and the 4090 lacks the VRAM for this model regardless.

Sources

  • RTX 5090 — the only consumer card with native NVFP4 support
  • RTX 4090 — still the used-market VRAM king for local models like Nemotron 3 Nano
  • RTX 3090 — cheapest 24GB entry point for running Nano-class models

Was this article helpful?