← All posts
Tutorial

MTP with LLAMA for FAST local inference

Scott ·

🚀 Local LLMs Just Got a Massive Upgrade: Native Multi-Token Prediction (MTP)

If you are running open-weights LLMs locally, stop what you are doing and look into native Multi-Token Prediction (MTP). It is a total game-changer for inference speeds.

Traditionally, to speed up local generation, you had to use Speculative Decoding—loading a completely separate, tiny "draft" model alongside your main model to guess the next few words. Managing two separate context windows, VRAM allocations, and mismatched vocabularies was a massive headache.

Native MTP changes the game. The draft capability is baked directly into the primary model's architecture during training. When you download an MTP-optimized GGUF (like Qwen 3.6), the lightweight drafting heads live right inside the same unified file.

The engine predicts the primary token while parallel heads glimpse 2 to 3 tokens ahead in a single forward pass. If the guesses are right, it accepts them all at once.

The results speak for themselves. On a standard baseline run without speculative flags, execution hits a typical ceiling. Turning on MTP pushes things into hyperdrive:

  • 📊 Baseline Speed: ~50 to 55 tokens per second
  • ⚡ Native MTP Speed: ~100 to 110 tokens per second!
  • 🎯 The Win: A ~2x pure speed multiplier with absolute zero loss in accuracy or model quality.

🛠️ How to Run It in llama.cpp

Make sure you are on a recent build that supports native MTP flags, and run your preferred setup.

💡 Option A: Single GPU Configuration

Perfect for a single-card setup running the Qwen 3.6 Dense 27B model:

.\llama-server.exe -hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q4_K_M --spec-type draft-mtp --spec-draft-n-max 2 -c 32768 -ctk q8_0 -ctv q8_0 -fa on

🔥 Option B: Dual-GPU Configuration (2x16GB VRAM Setup)

My exact setup running the ultra-fast Qwen 3.6 35B A3B Mixture-of-Experts (MoE) model. This structure forces a single-user execution lane (-np 1), handles layer distribution cleanly across both cards, quantizes the KV cache to save VRAM headroom, and leaves 1.5 GB of breathing room for Windows display traffic:

.\llama-server.exe -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_M --spec-type draft-mtp --spec-draft-n-max 3 -c 32768 -ctk q8_0 -ctv q8_0 -fa on --fit on --fit-target 1536 --jinja -np 1

Once running, look at your console metrics. Seeing an MTP draft acceptance rate consistently over 75% means you are generating massive text blocks with a fraction of the hardware compute.

If you are building agentic local systems or running interactive terminals like Claude Code, this level of throughput changes everything.

#ArtificialIntelligence #LocalAI #MachineLearning #LLMs #Nvidia #OpenSource #SoftwareEngineering