[ROCm][quant] INC: route w4a16-sym MoE through HybridW4A16 HIP path#929
Draft
mgehre-amd wants to merge 1 commit intogfx11from
Draft
[ROCm][quant] INC: route w4a16-sym MoE through HybridW4A16 HIP path#929mgehre-amd wants to merge 1 commit intogfx11from
mgehre-amd wants to merge 1 commit intogfx11from
Conversation
Wires INC (Intel Neural Compressor / auto-round) quantized models into the same HIP HybridW4A16 MoE path that compressed-tensors w4a16 already uses on ROCm. Auto-round emits its checkpoints in `auto_round:auto_gptq` packing (same on-disk layout as compressed-tensors `pack_quantized`), so the only INC-specific piece is registering the parameters under the GPTQ names (`w*_qweight` / `w*_scales` / `w*_qzeros`) that the standard FusedMoE expert-name mapping resolves; the conversion to ExLlama-shuffled `[E, N, K//8]` and the `HybridW4A16MoEExperts` modular-kernel install are reused from compressed-tensors. Verified on Strix Halo (gfx1151) with `Intel/Qwen3.5-35B-A3B-int4-AutoRound`: the `_rocm_C::fused_moe_wvSplitK_int4_gemm` kernel now drives the per-token MoE GEMMs on decode; non-MoE INT4 linears were already going through HybridW4A16LinearKernel via `choose_mp_linear_kernel`. Changes: - `vllm/platforms/rocm.py`: add `"inc"` to `supported_quantization` (the dispatcher behind it ultimately picks AWQ/GPTQ kernels through `choose_mp_linear_kernel`, so ROCm support is no longer unconditionally rejected at config validation). - `vllm/model_executor/layers/quantization/inc.py`: in `apply_awq_quant_layer` / `apply_gptq_quant_layer`, when the gate passes (`is_rocm`, 4-bit, sym, group_size>0, FusedMoE, non-marlin), return the new `INCHybridW4A16MoEMethod` instead of falling back to the generic `MoeWNA16Method`. - `vllm/model_executor/layers/quantization/inc_moe.py` (new): `INCHybridW4A16MoEMethod` registers GPTQ-named params, drops the sym `qzeros` (7-sentinel) before the kernel sees them, aliases to the names the helper expects, and installs the modular kernel. Originals are freed after the repack so weight memory stays at the checkpoint footprint instead of doubling. - `vllm/model_executor/layers/fused_moe/hybrid_w4a16_moe_helper.py` (new): shared `setup_hybrid_w4a16_moe(method, layer)` extracted from compressed-tensors; called by both backends so the conversion + modular-kernel install lives in one place. - `vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_wna16.py`: `_process_weights_hybrid_w4a16` now delegates to the shared helper. Bench (gfx1151, Intel/Qwen3.5-35B-A3B-int4-AutoRound, synthetic-mm 640x480, ISL/OSL=100/128, conc=1, --enforce-eager): decode 25.9 -> 36.4 tok/s (+40%), TPOT 38.7 -> 27.5 ms. Profile confirms `_rocm_C::fused_moe_wvSplitK_int4_gemm` is now on the decode hot path (was Triton MoeWNA16 before). Signed-off-by: Matthias Gehre <matthias.gehre@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wires INC (Intel Neural Compressor / auto-round) quantized models into the same HIP HybridW4A16 MoE path that compressed-tensors w4a16 already uses on ROCm. Auto-round emits its checkpoints in
auto_round:auto_gptqpacking (same on-disk layout as compressed-tensorspack_quantized), so the only INC-specific piece is registering the parameters under the GPTQ names (w*_qweight/w*_scales/w*_qzeros) that the standard FusedMoE expert-name mapping resolves; the conversion to ExLlama-shuffled[E, N, K//8]and theHybridW4A16MoEExpertsmodular-kernel install are reused from compressed-tensors.Verified on Strix Halo (gfx1151) with
Intel/Qwen3.5-35B-A3B-int4-AutoRound: the_rocm_C::fused_moe_wvSplitK_int4_gemmkernel now drives the per-token MoE GEMMs on decode; non-MoE INT4 linears were already going through HybridW4A16LinearKernel viachoose_mp_linear_kernel.Changes:
vllm/platforms/rocm.py: add"inc"tosupported_quantization(the dispatcher behind it ultimately picks AWQ/GPTQ kernels throughchoose_mp_linear_kernel, so ROCm support is no longer unconditionally rejected at config validation).vllm/model_executor/layers/quantization/inc.py: inapply_awq_quant_layer/apply_gptq_quant_layer, when the gate passes (is_rocm, 4-bit, sym, group_size>0, FusedMoE, non-marlin), return the newINCHybridW4A16MoEMethodinstead of falling back to the genericMoeWNA16Method.vllm/model_executor/layers/quantization/inc_moe.py(new):INCHybridW4A16MoEMethodregisters GPTQ-named params, drops the symqzeros(7-sentinel) before the kernel sees them, aliases to the names the helper expects, and installs the modular kernel. Originals are freed after the repack so weight memory stays at the checkpoint footprint instead of doubling.vllm/model_executor/layers/fused_moe/hybrid_w4a16_moe_helper.py(new): sharedsetup_hybrid_w4a16_moe(method, layer)extracted from compressed-tensors; called by both backends so the conversion + modular-kernel install lives in one place.vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_wna16.py:_process_weights_hybrid_w4a16now delegates to the shared helper.Bench (gfx1151, Intel/Qwen3.5-35B-A3B-int4-AutoRound, synthetic-mm 640x480, ISL/OSL=100/128, conc=1, --enforce-eager):
decode 25.9 -> 36.4 tok/s (+40%), TPOT 38.7 -> 27.5 ms.
Profile confirms
_rocm_C::fused_moe_wvSplitK_int4_gemmis nowon the decode hot path (was Triton MoeWNA16 before).
Purpose
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.