llama.cpp vision models: what the mmproj file is, and how to check it before you download 850 MB
TL;DR: llama.cpp runs a vision model from two GGUF files: the language model and a multimodal projector (mmproj). Upstream calls the projector “specific to each model architecture”, which means a mismatched projector is a broken setup you only find out about after the download. I checked four real files from the ggml-org Hugging Face repos on November 18, 2026: the projector is a GGUF in its own right, it records general.architecture = clip, and its clip.projector_type says which vision family it belongs to (gemma3 vs qwen2vl_merger). You can read all of that with a 30-line Python script and an HTTP range request — a few hundred kilobytes instead of 851 MB.
What you’ll be able to do:
- Tell a language-model GGUF from a projector GGUF by its metadata alone.
- Check which projector a repo ships, and which one your model needs, before downloading either.
- Read GGUF metadata without downloading the file, using a range request for the first few hundred kilobytes.
- Understand when the range has to be bigger than you expect, and why.
Honest take: this verifies the file, not the vision pipeline. Reading
clip.projector_typetells you the projector is for the right family; it does not tell you that the model answers questions about images correctly. Only running the model does that, and this article does not run one.
Two files, not one
The upstream tools/mtmd README states the arrangement plainly: “running a multimodal model typically requires two GGUF files: 1. The standard language model file. 2. A corresponding multimodal projector (mmproj) file, which handles the image encoding and projection.”
The reason is architectural. A vision model has a vision encoder plus a language model, and the encoder’s pre-processing and projection steps differ per family. llama.cpp keeps that part in a separate library — libmtmd, which replaced the older llava.cpp implementation — precisely so the multimodal half can evolve without destabilising the core.
The docs/multimodal.md page lists the flags and gives three example commands:
# simple usage with server
llama-server -hf ggml-org/gemma-3-4b-it-GGUF
# using local file
llama-server -m gemma-3-4b-it-Q4_K_M.gguf --mmproj mmproj-gemma-3-4b-it-Q4_K_M.gguf
# no GPU offload
llama-server -hf ggml-org/gemma-3-4b-it-GGUF --no-mmproj-offload
Two operational details in that page are worth remembering. “To load a model using -hf while disabling multimodal, use --no-mmproj” — a vision model can still be used as a text-only model, and some setups want exactly that. And “By default, multimodal projector will be offloaded to GPU” — on a machine where the projector does not fit in VRAM, --no-mmproj-offload is the switch that keeps it on the CPU instead of failing to load.
What the projector file actually is
It is a GGUF file. Same container format, same 24-byte header, same metadata block — a different job. I downloaded the first 256 KB of the mmproj that ships with Gemma 3 4B and read its header and metadata:
$ python3 gguf_meta.py mmproj-model-f16.gguf
GGUF version 3, 439 tensors, 16 metadata entries
general.architecture = clip
clip.projector_type = gemma3
clip.has_vision_encoder = True
clip.has_text_encoder = False
clip.has_llava_projector = False
clip.use_gelu = True
clip.vision.block_count = 27
clip.vision.embedding_length = 1152
clip.vision.feed_forward_length = 4304
clip.vision.projection_dim = 2560
clip.vision.attention.head_count = 16
clip.vision.image_size = 896
clip.vision.patch_size = 14
clip.vision.image_mean = [0.5, 0.5, 0.5]
clip.vision.image_std = [0.5, 0.5, 0.5]
general.architecture = clip is the marker that this is a projector rather than a language model. The language model in the same repository reports general.architecture = gemma3 and general.type = model. The projector’s clip.* keys describe the vision tower that was baked into it: how many blocks it has (27), how wide it is (1152), the image size it expects (896), the patch size (14), the normalisation constants, and the projection dimension it outputs (2560).
Those numbers are the point. Two different models can both call their projector mmproj-model-f16.gguf, and only one of them has a vision tower that matches the language model it was trained with.
The four files, measured
I pulled the file lists and sizes from the Hugging Face API for two ggml-org repositories on November 18, 2026, and read the metadata of each file’s first bytes:
File (repository ggml-org/...) | Bytes | general.architecture | general.type | general.file_type | Projector type |
|---|---|---|---|---|---|
gemma-3-4b-it-GGUF/gemma-3-4b-it-Q4_K_M.gguf | 2,489,757,856 | gemma3 | model | 15 | — |
gemma-3-4b-it-GGUF/mmproj-model-f16.gguf | 851,251,104 | clip | — | — | gemma3 |
Qwen2-VL-2B-Instruct-GGUF/Qwen2-VL-2B-Instruct-Q4_K_M.gguf | 986,046,944 | qwen2vl | model | 15 | — |
Qwen2-VL-2B-Instruct-GGUF/mmproj-Qwen2-VL-2B-Instruct-f16.gguf | 1,331,656,160 | clip | clip-vision | 1 | qwen2vl_merger |
general.file_type uses llama.cpp’s own quantization enum, where 15 is MOSTLY_Q4_K_M and 1 is MOSTLY_F16 — the same numbering used by the GGUF header check, which is where the format itself is explained.
Three numbers deserve attention:
- The projector is not a small sidecar. For Gemma 3 4B it is 851,251,104 bytes against a 2,489,757,856-byte model — 34.2 percent of the download. For Qwen2-VL 2B it is 1,331,656,160 bytes against a 986,046,944-byte model: the projector is 1.35 times the size of the language model.
- The two projectors are not interchangeable.
clip.projector_typeisgemma3in one file andqwen2vl_mergerin the other. Pairing the Qwen projector with the Gemma model gives you a file whose metadata says it belongs to a different vision family. - A mismatched pair fails late. Both files are individually valid GGUF files with valid headers, so nothing complains until llama.cpp tries to load the vision tower against the language model.
Total download for the Gemma 3 4B setup is 2,489,757,856 + 851,251,104 = 3,341,008,960 bytes, and that is only if you pick the right pair.
Checking a projector without downloading it
Hugging Face serves files over HTTP with range support, and the GGUF metadata lives at the very start of the file. That means a few hundred kilobytes is enough to answer “what is this file and is it the one I need”.
The range request is one line:
$ curl -sL -r 0-131071 \
"https://huggingface.co/ggml-org/gemma-3-4b-it-GGUF/resolve/main/mmproj-model-f16.gguf" \
-o mmproj-head.bin -w "%{http_code} %{size_download}\n"
206 131072
206 is the partial-content status; a 200 would mean you fetched the whole 851 MB instead.
The reader is short. A GGUF file starts with a 24-byte header — the four-byte magic GGUF, a uint32 version, and two uint64 counts for tensors and metadata entries — followed by the metadata key/value pairs, each a length-prefixed string key, a uint32 value type, and the value itself:
import struct
def read_string(buf, off):
(n,) = struct.unpack_from("<Q", buf, off)
off += 8
return buf[off:off + n].decode("utf-8", "replace"), off + n
def read_value(buf, off, vtype):
if vtype == 8: # string
return read_string(buf, off)
if vtype == 9: # array
(elem,) = struct.unpack_from("<I", buf, off); off += 4
(count,) = struct.unpack_from("<Q", buf, off); off += 8
vals = []
for _ in range(count):
v, off = read_value(buf, off, elem)
vals.append(v)
return vals, off
fmt = {0: "<B", 1: "<b", 2: "<H", 3: "<h", 4: "<I", 5: "<i",
6: "<f", 7: "<?", 10: "<Q", 11: "<q", 12: "<d"}[vtype]
(v,) = struct.unpack_from(fmt, buf, off)
return v, off + struct.calcsize(fmt)
buf = open("mmproj-head.bin", "rb").read()
magic, version, n_tensors, n_kv = struct.unpack_from("<4sIQQ", buf, 0)
off = 24
kv = {}
for _ in range(n_kv):
key, off = read_string(buf, off)
(vtype,) = struct.unpack_from("<I", buf, off); off += 4
val, off = read_value(buf, off, vtype)
kv[key] = val
print(magic, version, n_tensors, n_kv)
print(kv.get("general.architecture"), kv.get("clip.projector_type"))
On the real Gemma 3 projector that prints GGUF 3 439 16 and clip gemma3. On the Qwen2-VL projector it prints clip qwen2vl_merger. No model weights were downloaded for either check.
The range is not always enough
There is a catch, and it will bite you the first time you point this script at a language model.
The metadata block is variable in size. For both projector files, 128 KB was plenty — 16 and 27 metadata entries respectively. For the Gemma 3 language model, the same 128 KB range was not enough: the file has 40 metadata entries, including a 256k-entry tokenizer vocabulary, and the reader ran out of bytes. Asking for 8 MB was enough, and then it parsed cleanly:
$ python3 gguf_meta.py head-gemma-3-4b-it-Q4_K_M.gguf
GGUF version 3, 444 tensors, 40 metadata entries
general.architecture = gemma3
general.type = model
general.file_type = 15
So: start at 128 KB for a projector, and be ready to go to a few megabytes for a language model. If the script raises a struct.error or an IndexError, the answer is not a broken file — it is a truncated prefix.
Building the projector yourself
If a repository ships only the language model, upstream documents how to produce the projector yourself. The tools/mtmd README lists the models that convert_hf_to_gguf.py can build an mmproj for with the --mmproj flag: Gemma 3, SmolVLM, SmolVLM2, Pixtral 12B, Qwen 2 VL and Qwen 2.5 VL, Mistral Small 3.1 24B, InternVL 2.5 and 3, and MiniCPM-V 4.6. Older architectures have their own guides under docs/multimodal/.
Two caveats straight from the same README: the Gemma 3 1B variant has no vision support, so there is no projector to build for it; and for MiniCPM-V 4.6 you need the standard transformers v5.7.0+ checkpoint rather than a converted one.
And the README is blunt about the stability of all of this: multimodal support “is under very heavy development, and breaking changes are expected”. The flag names and the file layout can change between releases. That is an argument for checking the metadata of the file you actually have — which is exactly what the script above does — rather than trusting a filename.
Honest take: what this check does not tell you
- It does not verify the vision pipeline.
clip.projector_type = gemma3is a label, not a test. Only running the model against an image tells you the encoder and the language model actually work together. - It does not check the weights. A header and metadata parse says the container is well formed. File integrity is a separate question, and the safetensors header check covers the other container format; for GGUF the header and tensor-count check is the companion read.
- It does not tell you whether the projector is the right revision. Two files can both be
clip.projector_type = gemma3and still differ in which checkpoint they were converted from. The metadata narrows the field; it does not close it. - It says nothing about performance. Vision throughput depends on the encoder size and on whether the projector stays on the GPU, which is what
--no-mmproj-offloadis about.
What it does give you is the cheap half of the decision: which of the two files is the language model, which is the projector, which vision family the projector belongs to, how big each file is, and how many bytes you have to fetch to know all of that. Do that before the download, not after.
Sources
- Multimodal support in llama.cpp:
libmtmd,--mmproj,--no-mmproj,--no-mmproj-offload, and the list of pre-quantized vision models — github.com/ggml-org/llama.cpp - Multimodal support in llama.cpp (
tools/mtmd): what anmmprojfile is, how to obtain one withconvert_hf_to_gguf.py --mmproj, and the stability warning — github.com/ggml-org/llama.cpp LlamaFileTypequantization enum: 1 =MOSTLY_F16, 15 =MOSTLY_Q4_K_M— ggml-org/llama.cppgguf-py- ggml-org/gemma-3-4b-it-GGUF repository file list and sizes — Hugging Face API
- ggml-org/Qwen2-VL-2B-Instruct-GGUF repository file list and sizes — Hugging Face API
Last updated November 18, 2026. File sizes and metadata above were read from the Hugging Face API and from the first bytes of the published files on November 18, 2026, using HTTP range requests; no full model weights were downloaded and no inference was run. The upstream documentation was checked on the same date. No paid API, hosted service, or performance claim is involved.
Was this article helpful?
Thanks for the feedback — it helps improve future articles.
Need hands-on help?
I offer 1-on-1 technical consulting for local AI setup, GPU selection, and AI coding tool configuration — same topics covered on this site.
Book a session — $49 / hour →What self-hosting actually costs
Real cost breakdowns for self-hosted AI: hardware floors, power, maintenance hours, and the honest comparison against paying for it. No spam, unsubscribe anytime.