One flag
I run Qwen3.8 on three boxes at home. A 27B model on an RTX 5090 serves my coding agents. A 35B mixture-of-experts version on an RTX 5080 is the backup, and a 9B on an RTX 4060 handles small jobs.
This week a post went around claiming Ollama ran the same Qwen3.8 file about twice as fast as llama.cpp. Its explanation was that Ollama turns MTP on by default and stock llama.cpp leaves it off. I checked my own servers, and the two bigger ones had the MTP weights on disk doing nothing.
What MTP is
MTP stands for multi-token prediction. Qwen trains the model with one extra small layer, the MTP head, that predicts a few tokens past the next one.
At serving time it works as speculative decoding with a drafter that is built into the model:
- The MTP head drafts the next 2 or 3 tokens. This is cheap.
- The full model checks all the drafted tokens in one forward pass.
- Tokens the full model agrees with are kept. At the first disagreement, the rest are thrown out and the full model's own token is used.
The full model has the final say on every token, so the output follows the same distribution as without MTP. What changes is how many tokens you get per expensive forward pass.
Decoding on a single GPU is limited by memory bandwidth. The GPU spends most of each step reading weights, so checking 3 tokens costs about the same as producing 1, and each accepted draft token comes almost free.
Where it was hiding
My 5080 runs empero-ai/Qwen3.8-35B-A3B-Distill-GGUF at Q4_K_M under llama.cpp. Its startup log had this on every boot:
model has unused tensor blk.40.nextn.eh_proj.weight (size = 4718592 bytes) -- ignoring
nextn is the MTP layer. The GGUF ships it, and llama.cpp skips it unless you ask for it.
My 5090 was running philbert440/Qwen3.8-27B-W4A16-AWQ under vLLM. The repo has a separate model-mtp.safetensors file, 0.85 GB, with the MTP head kept in BF16. The model card gives the vLLM flag, and I had never turned it on.
Not every file has the head. The unsloth/Qwen3.5-9B-GGUF I had on the 4060 did not, so I moved that box to empero-ai/Qwen3.8-9B-Distill-GGUF, which does.
Results
Same three prompts on each box (a Laravel controller with tests, a Python log parser with tests, a B-tree explanation). One request at a time, 1500 output tokens, temperature 1.0, top-p 0.95, top-k 20. Throughput is output tokens divided by wall time.
Box / model / engine Setting Run 1 Run 2 Run 3 Avg
RTX 5090, 27B AWQ, vLLM (64K ctx) MTP off 57.3 60.3 59.3 59
MTP n=2 91.9 92.7 79.1 88
MTP n=3 91.6 98.9 108.6 100
RTX 5090, 27B IQ4_XS, llama.cpp (128K) MTP off 77.0 76.6 74.6 76
DFlash2 n=7 84.1 95.3 89.1 89
MTP n=3 117.0 120.0 88.5 108
RTX 5080, 35B-A3B Q4_K_M, llama.cpp MTP off 50.6 53.7 50.0 51
MTP n=2 73.0 74.4 57.3 68
MTP n=3 69.7 71.0 58.5 66
RTX 4060, 9B Q4_K_M, llama.cpp MTP off 43.4 44.2 44.0 44
MTP n=1 48.3 46.7 43.9 46
MTP n=3 38.7 28.7 29.0 32
On the 5090 under vLLM, per-position acceptance at n=3 was 0.74, 0.52 and 0.38. On average 2.6 tokens came out of each full forward pass instead of 1.
The gain shrinks as the card gets smaller. The 5080 cannot hold the whole 35B MoE in 16 GB, so some expert weights run on the CPU, and checking a batch of draft tokens there costs more. On the 4060, n=3 was slower than no MTP at all, and n=1 barely helped. My guess is that the small card runs out of compute before it runs out of memory bandwidth, so verifying extra tokens is no longer free. I have not profiled it to confirm.
DFlash2
DFlash2 is a separately trained block-diffusion drafter for Qwen3.8-27B. A LocalLLaMA post this week showed 248 tok/s on a 5090 with vLLM, and a public llama.cpp setup reports 212 to 258 on code.
My first try was under vLLM. The drafter needed its own VRAM, and on the 32 GB card that cut the KV cache until vLLM could only offer about 18K tokens of context. I backed it out.
Under llama.cpp it fit at 128K context. It ran 89 tok/s on my prompts, with about 2.5 tokens per forward pass against the 4.8 that setup reports. That setup used a patched llama.cpp fork and its own benchmark, and its numbers drop to 114 to 127 on open-ended prose. My prompts produce a lot of thinking text at temperature 1.0, which looks more like prose than code. MTP on the same llama.cpp server did better, so that is what I kept.
Switching engines
The llama.cpp numbers surprised me more than MTP did. Plain llama.cpp on an IQ4_XS GGUF ran 76 tok/s against 59 for vLLM on the AWQ build, on the same card. With MTP on both, llama.cpp still led, 108 to 100, and it fits 128K context where vLLM fit 64K.
Speed is not the only thing that matters for a coding agent, so I ran my eval: a six-step task where an agent builds out a Laravel landing page, with tests, a build and a page check after each step, and up to two fix attempts when something fails. The llama.cpp setup passed all six steps with no fixes in 542 seconds. The previous best on this eval was the 35B MoE on the 5080 at 708 seconds.
The 5090 now serves its main endpoint from llama.cpp. vLLM handled two requests at once and this setup handles one, which I can live with on a single-user box.
The flags
llama.cpp (build 11804):
--spec-type draft-mtp --spec-draft-n-max 3
vLLM:
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
Caveats
- This is one card per setup, three prompts, and single-request throughput. Your numbers will differ.
- Acceptance rate depends on the text. Predictable output like code boilerplate gets more accepted tokens than open-ended prose.
- Bigger draft counts are not always better. The best value was 3 on the 5090, 2 on the 5080 and 1 on the 4060.
- Check that your quant kept the MTP head. Some quantizers strip it to save space. In llama.cpp, look for
nextntensors in the startup log. In a safetensors repo, look formtp.tensor names in the index file.