TL;DR: Gemma 4 12B (June 3 2026, Apache 2.0) is the first mid-sized open model to read text, images, audio, and video in a single encoder-free pass — and it fits in ~7GB at Q4, so any 8GB GPU runs it. Image input works cleanly through Ollama; audio and video currently need llama.cpp’s llama-mtmd-cli or Transformers. Worth it if you want one local model instead of a Whisper-plus-LLM pipeline.

What you’ll have running after this guide:

• Gemma 4 12B answering text and image prompts on a single 8GB GPU through Ollama’s OpenAI-compatible API on port 11434.
• Audio transcription and Q&A working locally via llama-mtmd-cli, no Whisper install and no cloud calls.
• A repeatable image-description command you can drop into a document-OCR or media-tagging pipeline.

Google DeepMind shipped Gemma 4 12B on June 3 2026, and two days later — June 5 — released quantization-aware-training (QAT) checkpoints for it. What makes this release different from every other “multimodal” open model you’ve set up: there are no separate vision or audio encoders. Raw image patches and raw audio waveforms get projected straight into the language model’s embedding space. One model, one weights file, four input types. This guide gets it self-hosted, covers how to actually feed it images and audio, and is honest about where the local tooling still lags the model.

What “encoder-free” actually changes for you

Most local vision models — LLaVA, the older Qwen2-VL builds, PaliGemma — bolt a CLIP-style vision transformer onto an LLM. That means a second model file (an mmproj projector), extra VRAM, and a pipeline that can break independently. Gemma 4 12B drops the encoders.

The vision path is a 35M-parameter embedder: raw 48×48 pixel patches get projected to the model’s hidden dimension with a single matrix multiply, with a factorized X/Y coordinate lookup attaching spatial position. Audio is even simpler — the separate audio encoder (the 12 conformer layers used in Gemma 4’s E2B and E4B edge variants) is gone entirely. Raw 16kHz audio is sliced into 40ms frames of 640 floats each and projected linearly into the same space as text tokens.

The practical upshot: lower multimodal latency, and the whole model fine-tunes in one pass. The catch, which we get to below, is that local runtimes were built around the encoder-plus-projector model, so audio and video support in the tooling is newer and rougher than image support.

	Gemma 4 12B	Gemma 4 26B-A4B	Whisper-v3 + text LLM
Modalities	Text, image, audio, video	Text, image, audio, video	Audio→text, then text only
Params	11.95B dense	26B (4B active MoE)	1.5B + your LLM
VRAM (Q4)	~6.6GB	~15GB	~2GB + LLM VRAM
License	Apache 2.0	Apache 2.0	MIT + your LLM’s license
Context	256K	256K	n/a
Setup	One model	One model	Two models + glue code

The right-hand column is the honest alternative: for pure speech-to-text, faster-whisper is faster and more accurate than Gemma’s audio path. Gemma 4 12B wins when you want transcription and reasoning over the same input in one model — “what’s the action item in this voice memo?” rather than just the transcript.

Hardware and license check

Gemma 4 12B is Apache 2.0. That is worth stating plainly because it is unusual: earlier Gemma generations shipped under the Gemma Terms of Use, and Google’s own blog confirms 12B is “fully permissive Apache 2.0 license with no commercial restrictions.” You can self-host it commercially, modify it, and redistribute it. No MAU cap, no attribution string, no revenue trigger.

VRAM by quantization:

• Q4_K_M: ~6.6GB — fits an 8GB GPU (RTX 3060, RTX 4060, or a 16GB unified-memory Mac).
• Q8_0: fits 16GB cleanly with room for context.
• BF16 (full precision): ~24GB.

For a 12B model running your day-to-day multimodal work, the QAT Q4 build is the one to use. If you plan to run the BF16 build for fine-tuning or need multiple concurrent users, a rented GPU makes sense — a single 24GB card on RunPod handles the full-precision weights. For buying advice on 8–16GB cards that run the Q4 build locally, runaihome.com covers the consumer GPU options for local AI.

Step 1: Install and pull with Ollama

Ollama is the fastest path to a working endpoint. It is built on llama.cpp and GGUF, so the 12B runs the same way earlier Gemma 4 models did.

# Install (Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Pull the QAT build — ~7GB download
ollama pull gemma4:12b-it-qat

# Sanity check
ollama run gemma4:12b-it-qat "In one sentence, what makes you different from Gemma 3?"

Expected output (yours will vary in wording):

I process text, images, audio, and video directly in a single model without
separate encoders, which lowers latency and lets me run in about 7GB of memory.

Use the -it-qat tag, not the naive Q4. This matters — see the accuracy trap below. Confirm it loaded on the GPU:

ollama ps
# NAME                  SIZE     PROCESSOR    UNTIL
# gemma4:12b-it-qat     7.8 GB   100% GPU     4 minutes from now

If PROCESSOR shows CPU or a split, your GPU doesn’t have enough free VRAM — close other models or drop to a smaller context.

Step 2: The accuracy trap (real problem, real fix)

Here is the mistake that quietly wrecks Gemma 4 quality. The QAT checkpoints are trained to be quantized, but a naive Q4_0 conversion of them still loses several points of accuracy because of a scale mismatch. On the 26B-A4B variant, a naive Q4_0 conversion measured 70.2% top-1 accuracy, while Unsloth’s dynamic method recovered it to 85.6% — a 15-point swing — at the same memory footprint.

The fix: never hand-convert the QAT checkpoint to Q4_0. Use a properly-built GGUF. Ollama’s -it-qat tag already does this for you. If you pull GGUFs directly for llama.cpp, use Unsloth’s UD-Q4_K_XL builds (unsloth/gemma-4-12B-it-qat-GGUF) rather than a generic Q4_0. Same size on disk, materially better answers.

If you skipped this and pulled a plain gemma4:12b Q4 tag, re-pull the -it-qat variant. This is the single most common Gemma 4 self-hosting error.

Step 3: Passing image input

Ollama exposes an OpenAI-compatible API on localhost:11434. Images go in as base64. One rule from the model card: place image content before the text in your prompt for best results.

# Encode an image and send it
IMG=$(base64 -w0 receipt.jpg)
curl http://localhost:11434/api/chat -d '{
  "model": "gemma4:12b-it-qat",
  "messages": [{
    "role": "user",
    "content": "Extract the total and the date from this receipt.",
    "images": ["'"$IMG"'"]
  }],
  "stream": false
}'

For document OCR at scale, Gemma 4 supports a configurable visual token budget — 70, 140, 280, 560, or 1120 tokens per image. A higher budget preserves fine detail (small text on a scanned invoice) at more compute; a lower budget is faster for simple layouts. Through Ollama you get the default; for explicit control you drop to llama.cpp.

Step 4: Audio and video (where the tooling gets honest)

This is the part most breathless launch coverage skips: image input works through Ollama’s API today, but audio and video input go through llama.cpp’s multimodal CLI or Transformers. Ollama’s request format was built around the images field, and audio/video support for Gemma’s raw-waveform path is still catching up across runtimes.

For audio, use llama.cpp’s llama-mtmd-cli, which handles Gemma’s multimodal projector:

# Build llama.cpp with your backend (CUDA shown)
cmake -B build -DGGML_CUDA=ON && cmake --build build -j

# Transcribe + answer over a voice memo (audio goes AFTER the text)
./build/bin/llama-mtmd-cli \
  -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \
  --mmproj mmproj-gemma-4-12B-it.gguf \
  --audio memo.wav \
  -p "Summarize this voice memo and list any action items."

Note the input-placement rule flips for audio: audio content goes after the text. Two hard limits from the model spec: audio caps at 30 seconds per input, and video caps at 60 seconds at 1 frame per second (so ~60 frames). For longer media, chunk it and loop. Video is passed as a sequence of frames — extract frames with ffmpeg, then feed them as images ahead of your instruction.

For programmatic pipelines, Hugging Face Transformers exposes all four modalities directly and is the most complete path today if you need audio and video in production code rather than the CLI.

When NOT to use Gemma 4 12B

• Pure speech-to-text. If all you need is a transcript, faster-whisper (see our Whisper engine shootout) is faster, more accurate on long audio, and not capped at 30 seconds per call.
• Long audio or video as-is. The 30s audio / 60s video ceilings mean anything longer needs chunking logic. A dedicated STT model plus a text LLM handles a two-hour recording with less friction.
• You only ever send text. A text-only model at the same size (or the 26B-A4B MoE) gives you the same reasoning without carrying multimodal projection overhead. See the 26B QAT self-hosting guide for the speed-focused sibling.
• Best-in-class vision benchmarks. For pure image understanding at the top end, dedicated VLMs still lead — our open-source VLM guide compares them. Gemma 4 12B’s edge is breadth in one model, not peak vision accuracy.

Benchmarks and real use cases

On text reasoning, Gemma 4 12B posts MMLU Pro 77.2%, GPQA Diamond 78.8%, and AIME 2026 77.5% — strong for a 12B dense model, and community testing has it roughly matching the larger Gemma 4 26B on several tasks. Treat the multimodal numbers as directional; independent multimodal benchmarks were still thin as of July 2026.

Where the single-model design pays off in practice:

• Document Q&A over scanned PDFs — one model reads the page image and answers, no OCR-then-LLM handoff. Wire it into AnythingLLM for local RAG.
• Voice-memo triage — transcribe and extract action items in one call for short clips.
• Screenshot description pipelines — batch-tag images with a fixed visual token budget for consistent cost.

FAQ

Does Gemma 4 12B really run on 8GB VRAM? Yes, at Q4. The QAT build is ~6.6–7GB, leaving room for a modest context on an 8GB card. For long 256K-context sessions plus images, 12GB or 16GB is more comfortable.

Can I use audio input through Ollama directly? Not reliably as of July 2026. Ollama’s images field handles vision; for audio and video, use llama-mtmd-cli from llama.cpp or Hugging Face Transformers, which support Gemma’s raw-waveform and frame inputs.

Why not just use the plain gemma4:12b tag? A naive Q4_0 conversion of the QAT checkpoint loses accuracy. Pull the -it-qat tag (or Unsloth’s UD-Q4_K_XL GGUF) — same disk size, materially better answers.

Is it actually Apache 2.0, unlike older Gemma? Yes. Google’s launch blog confirms Gemma 4 12B ships under Apache 2.0 with no commercial restrictions — a change from the Gemma Terms of Use on earlier generations.

What are the audio and video length limits? Audio: 30 seconds per input. Video: 60 seconds at 1 frame/second. Chunk longer media and loop.

Sources

• Introducing Gemma 4 12B: a unified, encoder-free multimodal model — blog.google
• Gemma 4 12B: The Developer Guide — Google Developers Blog
• google/gemma-4-12B — Hugging Face model card
• Gemma 4 QAT — Unsloth Documentation
• unsloth/gemma-4-12B-it-qat-GGUF — Hugging Face
• Gemma 4 model overview — Google AI for Developers
• Gemma 4 12B: Benchmarks, VRAM & How to Run It — TECHSY