The claim
A post this week said HyperQwen runs Qwen3.8-27B at 1,035 tokens per second on a single RTX 3090. The repo is syv-ai/HyperQwen, Apache 2.0. It is a patch series against a pinned vLLM 0.31.0 plus a script that requantizes the model for a 24 GB card. It drafts with the DFlash2 drafter and a 40K-token draft vocabulary, and it can draft tokens straight out of the prompt when the answer repeats it.
The README's numbers are from the author's 3090 capped at 250 W:
- 1,035 tok/s is the total across 64 concurrent requests. A single request in that batch mode gets 46.
- 127 tok/s is the single-user mode.
- 381 tok/s is when the answer quotes its own prompt, such as a rewrite or code edit.
- The long-context modes trade speed for room: 150K context at about 95 to 100 tok/s, 240K at 67.
The README states all of this plainly. The tweet led with the aggregate number.
A day earlier I had moved my 5090 from vLLM to llama.cpp with MTP speculative decoding, at 108 tok/s and 128K context. A user report on the repo showed a 5090 at roughly 400 tok/s on code and 200 on prose with DFlash2. That was enough for me to try it.
Which cards
My other two boxes are out. The 27B model needs 24 GB, and the RTX 5080 has 16 and the RTX 4060 has 8. Smaller Qwen checkpoints for 12 and 16 GB cards are on the project's roadmap, not in the repo.
The 5090 has 32 GB, but about 6 GB of it was held by the Windows desktop and a video encoder that shares the card. The default GPU_UTIL=0.93 would not have fit, so I set it to 0.78. vLLM still sized the KV cache to 136,429 tokens, enough for one request at 131,072.
Setup
The compose file pulls a 9.5 GB image. A prepare step then downloads and requantizes the model on the CPU, about 23.6 GB on disk with the fast variant and the DFlash2 drafter. It ran while my existing server kept serving. I put the settings in a compose override instead of the .env file:
SPEC=dflash2
CTX=long
PREFIX_CACHE=1
VLLM_WSL2_ENABLE_PIN_MEMORY=1
GPU_UTIL=0.78
The WSL2 line is required on Docker Desktop. Without it vLLM aborts with RuntimeError: UVA is not available. On compute capability 12.0 the launcher turns its FP8 KV path off and logs that it did. First boot took a few minutes to compile.
Decode speed
Same three prompts as my MTP tests: a Laravel controller with tests, a Python log parser with tests, a B-tree explanation. One request at a time, 1,500 output tokens, temperature 1.0. I threw away the first run, as the README advises.
Engine on RTX 5090, 27B at 128K ctx Run 1 Run 2 Run 3 Avg
vLLM AWQ, MTP n=3 91.6 98.9 108.6 100
llama.cpp IQ4_XS, plain 77.0 76.6 74.6 76
llama.cpp IQ4_XS, DFlash2 n=7 84.1 95.3 89.1 89
llama.cpp IQ4_XS, MTP n=3 117.0 120.0 88.5 108
HyperQwen, DFlash2 206.1 165.7 153.7 175
The vLLM AWQ row was at 64K context. The rest were at 128K.
175 is below the user report of 400 on code, which came from a different client with 400-token outputs. Qwen's thinking text at temperature 1.0 reads more like prose than code, and prose was about 200 in that report. Against my own baseline it was 62% faster.
The eval
Tok/s is not what I use the box for. It serves coding agents, so the test that matters is my terra-flow eval: an agent builds out a Laravel landing page over six steps, and after each step the harness runs the tests, the build and a page check. Failures get up to two fix attempts.
The first attempt died partway. Something called the main endpoint, the on-demand waker started the llama.cpp container on top of HyperQwen, and Windows ran low on memory. I stopped the waker and ran it again.
llama.cpp MTP n=3 HyperQwen DFlash2
Steps passed 6/6 6/6
Fix attempts 0 0
Step times (s) 41 90 43 91 101 46 55 87 37 111 73 62
Agent time 412 s 425 s
Test and build time 105 s 132 s
Wall time 542 s 585 s
Model calls 57 57
Output tokens 45.8K 57.0K
Prompt tokens, not cached 53K 134K
Prompt tokens, cached 573K 544K
Both passed every step without a fix. HyperQwen took 43 seconds longer.
Inside the eval HyperQwen still decoded faster: 57K output tokens in 425 seconds is about 134 tok/s of agent time, against 111 for llama.cpp. Two things ate that lead. On the same task it wrote 11K more tokens, and at temperature 1.0 that is partly luck. And two and a half times as many of its prompt tokens missed the cache and had to be processed. Every agent call sends the whole conversation back, so reading prompts is a large share of the work, and speculative decoding does nothing for it.
The extra 27 seconds of test and build time is Laravel running PHPUnit and Vite. The model is not involved there, and the machine was short on memory.
What I kept
llama.cpp with MTP stays on the 5090. One run per engine cannot separate a 43-second gap from noise, and a tie does not pay for what HyperQwen costs to run: a vLLM fork pinned to one version, 22 GB more weights, and a second compose project my GPU arbiter does not manage.
I kept the image and the prepared model. Two cases would change the answer. One is batch jobs with many requests in flight, where the README's numbers come from. The other is edit-heavy work where the answer repeats the prompt, which is what DFLASH_TOKENS=15 is built for.
Caveats
- One eval run per engine. Temperature 1.0 makes output length vary run to run.
- Three bench prompts, single requests, one card.
- My 5090 runs at its 575 W default limit and shares about 6 GB of VRAM with other work. The README numbers are from a 3090 capped at 250 W.
- I ran HyperQwen in its
singlemode with DFlash2 at 128K. Its MTP mode at long context, which the README recommends past 64K, I did not test.