Qwen3.8-27B Benchmarks: Throughput, Latency and KV Cache
Hello, I'm Yongrae Cho. I work on AI infrastructure R&D on the AIEEV development team.
Air API now serves the Qwen3.8-27B-BF16 model. The Air API Playground already provides a test environment, so you can try the model on your own workloads and prompts right now without setting up infrastructure or a serving engine. This post starts with the main characteristics of Qwen3.8-27B and its public benchmark scores. It then shares the throughput, latency and memory behavior we measured on a single GPU, and closes with how to use those results when selecting and validating a model for a real service.
What kind of model is Qwen3.8-27B?
Qwen3.8-27B is a multimodal AI model that covers a wide range of work, from conversation and document understanding to code writing and complex multi-step tasks. The Qwen Team at Alibaba Group released it on August 14, 2026. It builds on the Qwen3.5 architecture with stronger coding, professional work, research and long-horizon task performance. It understands images and video in addition to text, and it can adjust reasoning depth to the complexity of the task. Within three weeks of release it had drawn roughly 13.8K likes and more than 5.25 million downloads on Hugging Face. The Qwen Team released a BF16 model with the original weights alongside an FP8 checkpoint using block-wise quantization, and reported that the FP8 version keeps performance close to BF16 on major benchmarks while using less memory. Air API serves the BF16 model. To see how the difference between the two shows up in an actual inference setup, we ran a quantization comparison of BF16 and FP8 throughput and memory use, covered later in this post. [3][4]
This is not the first Qwen model we have introduced. Air API already serves several open-weight models including Qwen3.5-9B, Qwen3.6-35B-A3B and Qwen3-TTS, and we covered the cheapest way to run Qwen in an earlier post. Adding Qwen3.8-27B widens the range you can choose from by use case and scale. The table below summarizes the specifications from the official model card. These values are the ceilings the model supports. Actual throughput and time to first token depend on request length, the number of concurrent requests and the serving configuration.
Item | Specification |
Model served | Qwen3.8-27B-BF16 (the Air API model covered in this post) |
Parameters | 27 billion |
Weights | BF16 (16-bit floating point), listed size 51.8 GiB |
Language decoder | 64 layers (48 gated DeltaNet + 16 output-gated full attention) |
Context length | 256K by default, up to 1M with configuration |
Supported input | Text, image, video |
Next, an external comparison. The figure below compares Qwen3.8-27B with Nemotron 3.5 Lightning and Gemma 4 31B across three categories on OpenRouter: intelligence, coding and agentic tasks. The scores are not accuracy percentages. They are the composite external capability scores shown on the OpenRouter comparison view. Qwen3.8-27B scores 52 on intelligence, 68 on coding and 51 on agentic tasks, higher than both comparison models in all three categories. Coding and agentic tasks stand out in particular. Answer accuracy still needs separate validation, but this is enough reason to shortlist the model for services where complex task execution matters.
Qwen3.8-27B combines 48 gated DeltaNet layers with 16 output-gated full attention layers. All 64 layers include a dense FFN, and the FP8 checkpoint uses the same architecture as BF16 with only the weight precision changed. The table and figure below also list the models we used to compare architecture and performance. The listed size reflects the parameter scale of each model and differs from the total memory required to actually run it. The input types are the specifications each model itself supports, which should be read separately from the input range Air API currently offers.
The model marked as the serving baseline in the table is used only as a reference point for locating relative throughput in the single-GPU experiments that follow. [3][4][7][8]
Model | Parameters | Weight format and listed size | Layer and block composition | Max context length | Multimodal input |
Qwen3.8-27B-BF16 | 27 billion | 16-bit floating point (BF16), 51.8 GiB | 64 layers: 48 gated DeltaNet + 16 output-gated full attention | 256K 1M with config | Yes |
Qwen3.8-27B-FP8 | 27 billion | 8-bit floating point (FP8), 28.7 GiB | Same as the BF16 version | 256K 1M with config | Yes |
Nemotron 3.5 Lightning | 30 billion 3 billion active | 16-bit floating point (BF16), 61.3 GiB | 52 blocks: 23 Mamba2 + 23 mixture of experts + 6 full attention | 1M | No |
Gemma 4 31B (serving baseline) | 31 billion | NVIDIA 4-bit floating point (NVFP4), 30.4 GiB | 60 layers: 50 sliding window attention + 10 full attention | 256K | Yes |
What we measured on Qwen3.8-27B
Now to the numbers we measured ourselves. These tests look at how throughput, latency and memory behavior change under actual serving. The table below lists the experiment conditions. Within each experiment we kept every setting other than the variable under comparison as close to identical as possible.
Category | Setting and measurement method |
Hardware | 1x NVIDIA RTX PRO 6000 96GB |
Models tested | Air API model: Qwen3.8-27B-BF16 Quantization comparison: Qwen3.8-27B-FP8 Serving baseline: Gemma 4-31B-IT-NVFP4 |
Serving engine | vLLM 0.27.1 |
Weights and cache | Qwen BF16: 16-bit weights and cache Qwen FP8: 8-bit weights, 16-bit cache Baseline: NVFP4 weights, 8-bit cache |
Attention execution path | Both Qwen checkpoints use the same execution path and the baseline uses a different one |
Common conditions | GPU memory utilization cap 90%, synthetic random requests, random seed 42, all requests submitted at once |
Context length sweep | Max context length 2K to 128K, max-num-batched-token 8K, concurrency 32, 64 requests total |
max-num-batched-token sweep | Max context length 32K, max-num-batched-token 1K to 32K, concurrency 16, 32 requests total |
Request composition | Input to output token ratio 8:2, with actual request length spanning 10% to 100% of the configured maximum |
max-num-batched-tokens in the table is the maximum number of tokens vLLM can process in a single scheduling step. Raising it lets more tokens run at once, but memory use and request latency shift along with it. A few things to keep in mind before reading the results.
In the experiments with longer max context lengths, the actual request lengths grew as well.
Total token throughput is the metric reported by the benchmarking tool. It does not mean generated-token speed alone.
This is a synthetic load test with many requests running concurrently, so latency here can differ from a typical chat setting or from Air API performance guarantees.
Gemma 4 differs from the Qwen models in weight precision, cache precision and attention execution path. We do not read it as a fully controlled one-to-one comparison.
Results
The three main findings are as follows.
As max context length grows, the gap between Qwen BF16 and the baseline narrows, and at 128K Qwen comes out ahead.
Changing max-num-batched-token from 4K to 32K barely moves throughput, which holds between 3,475 and 3,505 tokens per second. Across that range it sits 17% to 19% below Gemma 4.
On memory, the key-value cache per request grows with context length, so the number of requests that can be served concurrently drops quickly.
Experiment 1. Max context length from 2K to 128K
First, the context length sweep. We varied max context length (max_model_len) from 2K to 128K and measured request throughput, total token throughput, time to first token, inter-token latency and end-to-end latency. The actual request length distribution was configured to grow with max context length, so the latency increase reflects both the setting and the longer requests.
Qwen BF16 peaks at 32K with 3,856 tokens per second, and Qwen FP8 peaks at 64K with 4,707 tokens per second. At 64K and 128K, Qwen FP8 records the highest throughput of the three checkpoints. Looking only at the Qwen BF16 checkpoint, the one Air API serves, total token throughput runs 24.2% to 27.3% below the serving baseline between 8K and 32K, and the gap narrows to 13.5% at 64K. At 128K, Qwen BF16 is 7.3% higher instead. The baseline leads at mid-range context lengths, but the difference shrinks as context grows and Qwen BF16 leads in the longest range we tested. At 128K, Qwen FP8 has 21.8% higher throughput than BF16 and roughly 25% shorter mean time to first token. Mean end-to-end latency is also shorter at 485 seconds against 554 seconds for BF16. Mean inter-token latency, on the other hand, is shorter for Qwen BF16 at 218 milliseconds against 326 milliseconds for FP8. That is why first response and generation speed have to be read separately.
Experiment 2. max-num-batched-token from 1K to 32K
In the next experiment we fixed max context length at 32K and concurrency at 16, and varied max-num-batched-token from 1K to 32K. Gemma 4 did not run at the 1K and 2K settings. Those results do not mean a measured performance of zero. The run stopped during checkpoint loading because the setting did not meet the minimum required for image token preprocessing.
Qwen BF16 stays within 1% throughput variation from 4K to 32K, at 3,475 to 3,505 tokens per second. Qwen FP8 reaches 20.0% and 20.3% higher throughput than BF16 at 4K and 8K, but falls 17.0% and 22.8% lower at 16K and 32K. In this environment, that makes 4K to 8K the range to start from when using FP8.
From 4K to 32K, Qwen BF16 holds 80.9% to 83.4% of the serving baseline throughput. The gap per condition is roughly 17% to 19%, but the two checkpoints differ in cache precision and attention execution path, so this cannot be read as an effect of model architecture alone. The baseline should be treated as a reference for relative position.
max-num-batched-token | Qwen BF16 | Qwen FP8 | Gemma 4 |
1K | 3,204 / 25.0s | 3,911 / 20.0s | N/A (did not run) |
2K | 3,284 / 24.9s | 4,057 / 15.4s | N/A (did not run) |
4K | 3,502 / 14.9s | 4,202 / 12.6s | 4,331 / 10.9s |
8K | 3,477 / 15.6s | 4,182 / 13.9s | 4,253 / 12.6s |
16K | 3,475 / 17.3s | 2,883 / 23.8s | 4,221 / 11.9s |
32K | 3,505 / 18.9s | 2,705 / 31.7s | 4,205 / 12.3s |
[ Table. By max-num-batched-tokens setting: 'total token throughput' / 'mean time to first token' ]
Experiment 3. Key-value cache headroom
For the memory measurements on Qwen3.8-27B-BF16 we calculated available key-value cache, the number of tokens that fit in the key-value cache, the theoretical concurrent capacity for maximum-length requests and the cache per request. Concurrent capacity is an arithmetic value, the cache token count divided by the configured max context length. It is not a real service concurrency figure or a performance guarantee.
Available key-value cache stayed nearly flat at 31.25 to 31.37 GiB. The cache required per request, on the other hand, grew from 0.43 GiB at 2K context to 8.33 GiB at 128K and 16.33 GiB at 256K. Theoretical concurrent capacity fell from 72.78 to 1.92, a 97.4% drop. Supporting a long context and serving many long requests at the same time are two separate operational problems.
Max context length (max_model_len) | Available key-value cache (GiB) | Key-value cache tokens | Theoretical concurrent capacity | Key-value cache per request (GiB) |
2K | 31.35 | 149,048 | 72.78 | 0.43 |
4K | 31.35 | 223,573 | 54.58 | 0.57 |
8K | 31.35 | 315,632 | 38.53 | 0.81 |
16K | 31.37 | 397,463 | 24.26 | 1.29 |
32K | 31.35 | 447,146 | 13.65 | 2.30 |
64K | 31.37 | 476,956 | 7.28 | 4.31 |
128K | 31.25 | 491,896 | 3.75 | 8.33 |
256K | 31.35 | 503,531 | 1.92 | 16.33 |
[ Table 4. By max context length. On Qwen3.8-27B-BF16, max-num-batched-token was fixed at 8K and max concurrent sequences at 256. For 256K we checked memory capacity only and did not include it in the performance experiments. ]
max-num-batched-token | Available key-value cache (GiB) | Key-value cache tokens | Theoretical concurrent capacity | Key-value cache per request (GiB) |
1K | 31.43 | 447,829 | 13.67 | 2.299 |
2K | 31.39 | 447,146 | 13.65 | 2.300 |
4K | 31.37 | 447,146 | 13.65 | 2.298 |
8K | 31.35 | 447,146 | 13.65 | 2.297 |
16K | 30.13 | 429,397 | 13.10 | 2.300 |
32K | 27.10 | 386,389 | 11.79 | 2.299 |
[ Table 5. By max-num-batched-token. On Qwen3.8-27B-BF16, max context length was fixed at 32K and max concurrent sequences at 256. ]
In this experiment, available KV cache stayed at roughly 31.4 GiB while max-num-batched-tokens ran from 1K to 8K. Raising it from 8K to 32K, however, cut available cache from 31.35 GiB to 27.10 GiB, a 13.6% drop, and theoretical concurrent capacity fell from 13.65 to 11.79. Because max context length was fixed at 32K, the cache required per request held steady at about 2.30 GiB. So max-num-batched-token should not be raised on throughput alone. Set it with KV cache headroom and the number of concurrent requests in view as well.
BF16 against FP8
For the quantization effect we compare only Qwen3.8-27B-BF16 and Qwen3.8-27B-FP8, which share an architecture. The two checkpoints have the same 27 billion parameters, the same 64-layer composition and the same attention execution path. Weight precision is 16-bit on the BF16 version and 8-bit on the FP8 version, and the FP8 version keeps a 16-bit key-value cache. Performance here means throughput and latency. Answer quality differences are not measured in this experiment.
The FP8 checkpoint's listed weight size is 28.7 GiB against 51.8 GiB for BF16, 44.6% smaller. Averaged plainly across the seven context lengths, FP8 runs about 3,708 tokens per second against about 2,935 for BF16, 26.3% higher. Weight size drops by nearly half while average throughput rises by about a quarter, so the throughput gain is less dramatic than the size reduction. The gain also varies widely by context length, from 17.2% to 53.7%. This size covers weights only. It is not total GPU memory use, which includes the key-value cache and serving engine memory.
FP8 throughput peaks at 64K with 4,707 tokens per second. At 128K, mean time to first token is about 350 seconds against about 467 seconds for BF16, 25% shorter. At max-num-batched-token of 16K and 32K, on the other hand, it is slower than BF16.
Closing
Laid over each other, the three experiments show settings that interact. max-num-batched-token gives the shortest time to first token at 4K, 14.9 seconds, and raising it further moves throughput by less than 1% while available cache drops from 31.35 GiB to 27.10 GiB. Past 32K, max context length saturates throughput while the cache per request keeps growing. Whether to keep a long context open or to accept many concurrent requests is a choice between the two. The gap against Gemma 4 is not a verdict on which model is better. Cache precision and attention execution path differ, so the conditions are not controlled. The number does not mean Qwen is that much slower. It is a scale for gauging roughly where throughput sits. The public scores we looked at earlier are quality-side values and cannot be placed next to throughput either. So the order of validation depends on the service. If answer quality and multi-step work matter, running Qwen3.8-27B on your own data first is the faster path. If throughput and time to first token on mid-length requests come first, the baseline results belong in the picture too. It is not the fastest model under every condition, but throughput holds steady in the ranges used in practice and leads at long context. That is the conclusion of this round of measurements. If you would rather check the model first without touching these settings, you can call it directly from Air API.
With Air API you can connect Qwen3.8 to your application without preparing a GPU server and a serving engine. It supports an OpenAI-compatible call format, so existing integration code can start by swapping the endpoint and API key, and you can check prompts and responses in the Playground before applying it to your product.
🖥️ Coding assistance: generating, editing and explaining code, drawing on the model's coding scores
🤖 Agentic work: automation flows that chain several steps, validated against real data
📄 Long document processing: work where throughput and latency by context length need to be checked on actual documents
If answer accuracy and complex task execution matter to your service, try Qwen3.8-27B-BF16 on your own data first. You can start with Qwen3.8-27B-BF16 on Air API today.
👉 Try it now: Qwen3.8-27B
👉 Air API guide: Air API docs
.
.
.
References
[1] Air API docs - getting started flow and Playground
[2] Qwen official GitHub - developer and release date
[3] Qwen3.8-27B official model card - key features, specifications, Hugging Face activity
[4] Qwen3.8-27B-FP8 official model card - FP8 quantization method
[5] OpenRouter: Qwen vs Nemotron - original source for Figure 1
[6] OpenRouter: Qwen vs Gemma - original source for Figure 1
[7] Gemma 4 31B official model card - max context length and input types
[8] Nemotron 3.5 Lightning official model card - max context length and input types
[9] OpenAI-compatible API guide - integrating existing code
