question about sliding window attention

[ZelinMa557]

Thanks for the great work!

I notice that dflash drafters for qwen3.6 use sliding window attention, I'm curious about the reason behind it.

Is that for lower draft latency, or sliding window attention can increase accept length?

If I force to use full attention for sliding window layers, will the accept length drop?

[jianc99]

Yes, because Qwen3.6 models are hybrid models with a 1:4 ratio of full-attention layers to linear-attention layers. If the draft model uses 5 full-attention layers, the draft cost becomes large when the context length is very long, for example 50K tokens in agentic workloads, which hurts the overall speedup.

Therefore, we decided to use interleaved SWA layers in the draft model. In our tests, this does not hurt the acceptance length, while it helps improve long-context inference acceleration.

Since the draft model is trained with sliding-window context for the SWA layers, forcing those SWA layers to use full attention will cause the acceptance length to drop.

[Sylvan820]

Yes, because Qwen3.6 models are hybrid models with a 1:4 ratio of full-attention layers to linear-attention layers. If the draft model uses 5 full-attention layers, the draft cost becomes large when the context length is very long, for example 50K tokens in agentic workloads, which hurts the overall speedup.

Therefore, we decided to use interleaved SWA layers in the draft model. In our tests, this does not hurt the acceptance length, while it helps improve long-context inference acceleration.

Since the draft model is trained with sliding-window context for the SWA layers, forcing those SWA layers to use full attention will cause the acceptance length to drop.

So my understanding is that you added SWA to the mask of the first 4 DFlash layers, right?

[Sylvan820]

是的，因为 Qwen3.6 模型是混合模型，其中全注意力层与线性注意力层的比例为 1:4。如果草稿模型使用 5 个全注意力层，那么当上下文长度非常长时（例如在代理工作负载中达到 5 万个 token），草稿成本就会很高，从而影响整体加速。

因此，我们决定在模型草稿中使用交错式SWA层。在我们的测试中，这不会影响接受长度，同时有助于提高长上下文推理的速度。

由于草稿模型使用滑动窗口上下文对 SWA 层进行训练，强制这些 SWA 层使用完全注意力会导致接受长度下降。

Hi authors, I’d like to ask whether any modifications are needed on the SGLang side to support this configuration, or whether the latest SGLang already supports dflash SWA layers.