[AMD/ROCm] kimik2.5 int4 mi355x update

[seungrokj]

waiting for the optimized upstream docker image.

Regards,
Seungrok

[github-actions]

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

• https://cookbook.sglang.io/
• https://docs.vllm.ai/projects/recipes/en/latest/

If it is not, please create a PR first before we can merge your PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

• https://github.com/vllm-project/recipes
• https://github.com/sgl-project/sgl-cookbook

[functionstackx]

@seungrokj can u add in ur perf-changelog that VLLM_ROCM_USE_AITER=1 works now due to PR vllm-project/vllm#35641

[functionstackx]

@seungrokj any updates on this PR?

[functionstackx]

@claude create a new PR that ports this PR over & uses vllm/vllm-openai-rocm:v0.18.0 image

[Klaud-Cold]

Claude finished @functionstackx's task in 2m 59s —— View job

---

Porting PR #909 with vllm/vllm-openai-rocm:v0.18.0 image

• Gather context from PR [AMD/ROCm] kimik2.5 int4 mi355x update #909 changed files
• Create new branch from main with ported changes
• Update image to vllm/vllm-openai-rocm:v0.18.0
• Create new PR

Summary

Created PR #950 which ports all changes from this PR with the resolved upstream docker image:

• Image: v0.15.1 → v0.18.0 in amd-master.yaml
• AITER MLA: Added export VLLM_ROCM_USE_AITER=1
• Benchmark script: Added --max-num-seqs 256, removed --disable-log-requests
• perf-changelog: Added entry documenting all changes

#950

--- |

[seungrokj]

hi @functionstackx I tested a few thing to figure out non-deterministic behavior of kimik2 fp4 case. And now it's resolved. So now will work on "int4" model. Or, I'll update on top of mr. Klaud's work on #950

[functionstackx]

@seungrokj feel free to continue working on this! appreipcate ur help!

[functionstackx]

i started an klaud cold PR cuz i thought u were busy with other tasks and didnt wanna delay adding kimi k2.5 int4 mi355 AITER which already is an easy win cuz it has great perf improvmenets

[seungrokj]

i started an klaud cold PR cuz i thought u were busy with other tasks and didnt wanna delay adding kimi k2.5 int4 mi355 AITER which already is an easy win cuz it has great perf improvmenets

Sure will work on this today!

[functionstackx]

hi @seungrokj

since #909 already passed validation & improvements is easy win & we ideally want to show these improvements on the frontend ASAP, i am gonna merge #909 . for any additional changes, can u build on top of #909

[seungrokj]

@functionstackx yes if we are just using tp8. Then #909 should be solid. It also aligned with internal measurement on
vllm/vllm-openai-rocm:v0.18.0
img.

[functionstackx]

@seungrokj do u see anything better with TP4? if TP4 is on the pareto frontier, feel free to add it

[seungrokj]

@functionstackx based on fp4 case int4 (same memory footprint) could have better tput/gpu. On testing internally and if it looks good then will raise a subsequent PR.

[functionstackx]

thanks! looking forward to ur follow up PR on if tp4 is better or not