syv-ai runs Qwen3.8-27B on one 24 GB RTX 3090

The syv-ai/qwen38-27b-rtx3090 repository serves Qwen3.8-27B on a single 24 GB RTX 3090 using vLLM, with a 150,000-token context window and an OpenAI-compatible API. It was created on 15 August 2026 and has 1,077 stars and 139 forks on GitHub as of the trending sweep on 3 September, all inside its first 19 days.

What is packaged

The README documents two run modes served from the same prebuilt image. The batch profile targets API backends with many concurrent requests. The single-user profile targets one or a few people chatting. Both use the same install and differ only in which launch script runs. The image is built and pushed to GitHub Container Registry on every commit, and the first start pulls it at 9.5 GB then downloads and requantizes the model at about 20 GB into a local models directory.

The reported throughput numbers

syv-ai reports the results were measured with vLLM's serving benchmark on an RTX 3090 held at a 250 W power limit. All numbers below are from the repository's README and its benchmark documentation, and the project describes each configuration in enough detail that the measurements can be reproduced with the included scripts.

Aggregate at 64 concurrent users, 128-token inputs and 512-token outputs, the README reports around 1,035 tokens per second steady-state decode and 948 end-to-end, rising to about 1,222 and 1,042 with all layers running at int8. Single-stream decode at C1 on realistic prompts is reported at 46 tokens per second before speculative decoding is turned on.

With MTP speculation, the project reports 121 tokens per second at default sampling and 120 greedy at a 64k context, moving to 96 and 102 at the full 150k context. A second speculative path called DFlash2 is reported at 127 default and 130 greedy on the same 64k configuration, and 381 tokens per second at 25k context when a request keeps reproducing content that is already in the prompt.

How syv-ai says the speedups work

The README credits three settings for most of the gain. MTP speculation runs four cheap drafts with a draft vocabulary calibrated to the model's own outputs and an int4 language-modelling head, and verifies against split-KV attention. DFlash2 swaps the trained MTP head for a block drafter that proposes seven tokens in one forward pass instead of chaining four, backported from a vLLM pull request. Prefix caching keeps both the attention KV cache and the recurrent state for a document that has already been sent, which drops time-to-first-token on a 25k-document second turn from 22.4 seconds to 0.56 seconds according to the benchmark table.

Two things are worth taking seriously. The first is the hardware. An RTX 3090 sells second-hand at a small fraction of a rented H100 hour, and the repository puts a 27-billion-parameter reasoning model with a 150k context inside that budget. A team that had assumed a small in-house model needed a data-centre GPU should look at the included benchmark script and rerun it before committing to the assumption.

The second is that every figure in the README is the maintainer's own reporting. The repository is 19 days old, and none of the numbers here has been replicated by a third party. The included bench script is the point where a serious buyer would start, and the maintainer names the exact environment variables and the exact power limit needed to reproduce them, which is more disclosure than most vendor benchmarks come with.

Source

• syv-ai/qwen38-27b-rtx3090 on GitHub