[DNM - vllm 0.18 is bugged, need to wait till 0.19] Add MiniMax M2.5 NVFP4 benchmark for B200 vLLM (TP4, TP2) #952
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| source "$(dirname "$0")/../benchmark_lib.sh" | ||
|
|
||
| check_env_vars \ | ||
| MODEL \ | ||
| TP \ | ||
| CONC \ | ||
| ISL \ | ||
| OSL \ | ||
| MAX_MODEL_LEN \ | ||
| RANDOM_RANGE_RATIO \ | ||
| RESULT_FILENAME | ||
|
Check failure on line 13 in benchmarks/single_node/minimaxm2.5_fp4_b200.sh
|
||
|
|
||
| if [[ -n "$SLURM_JOB_ID" ]]; then | ||
| echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME" | ||
| fi | ||
|
|
||
| hf download "$MODEL" | ||
|
|
||
| nvidia-smi | ||
|
|
||
| export TORCH_CUDA_ARCH_LIST="10.0" | ||
| export PYTHONNOUSERSITE=1 | ||
|
|
||
| SERVER_LOG=/workspace/server.log | ||
| PORT=${PORT:-8888} | ||
|
|
||
| # EP support: conditionally enable expert parallel based on EP_SIZE env var | ||
| if [ "${EP_SIZE:-1}" -gt 1 ]; then | ||
| EP=" --enable-expert-parallel" | ||
| else | ||
| EP=" " | ||
| fi | ||
|
|
||
| # Start GPU monitoring (power, temperature, clocks every second) | ||
| start_gpu_monitor | ||
|
|
||
| set -x | ||
| vllm serve $MODEL --host 0.0.0.0 --port $PORT \ | ||
| --tensor-parallel-size=$TP \ | ||
| --gpu-memory-utilization 0.90 \ | ||
| --max-model-len $MAX_MODEL_LEN \ | ||
| --max-num-seqs $CONC \ | ||
| --no-enable-prefix-caching \ | ||
| --compilation_config.pass_config.fuse_allreduce_rms true \ | ||
| --trust-remote-code${EP} > $SERVER_LOG 2>&1 & | ||
|
|
||
| SERVER_PID=$! | ||
|
|
||
| # Wait for server to be ready | ||
| wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID" | ||
|
|
||
| pip install -q datasets pandas | ||
|
|
||
| run_benchmark_serving \ | ||
| --model "$MODEL" \ | ||
| --port "$PORT" \ | ||
| --backend vllm \ | ||
| --input-len "$ISL" \ | ||
| --output-len "$OSL" \ | ||
| --random-range-ratio "$RANDOM_RANGE_RATIO" \ | ||
| --num-prompts $(( CONC * 10 )) \ | ||
| --max-concurrency "$CONC" \ | ||
| --result-filename "$RESULT_FILENAME" \ | ||
| --result-dir /workspace/ \ | ||
| --trust-remote-code | ||
|
|
||
| # After throughput, run evaluation only if RUN_EVAL is true | ||
| if [ "${RUN_EVAL}" = "true" ]; then | ||
| run_eval --framework lm-eval --port "$PORT" --concurrent-requests $CONC | ||
| append_lm_eval_summary | ||
| fi | ||
|
|
||
| # Stop GPU monitoring | ||
| stop_gpu_monitor | ||
| set +x | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,11 @@ | ||
| - config-keys: | ||
| - minimaxm2.5-fp4-b200-vllm | ||
| description: | ||
| - "Add MiniMax M2.5 NVFP4 single-node B200 vLLM benchmark (TP4, TP2)" | ||
| - "Uses vllm/vllm-openai:v0.18.0 image with --no-enable-prefix-caching" | ||
| - "Concurrency 4-64, all three seq-len configs (1k1k, 1k8k, 8k1k)" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX | ||
|
Check warning on line 7 in perf-changelog.yaml
|
||
|
Contributor
There was a problem hiding this comment. 🟡 The new Extended reasoning...The The specific code path is straightforward: the diff shows the new entry prepended to Existing code does not prevent this because The impact is limited to changelog traceability: anyone reviewing the changelog to understand when or why Step-by-step proof:
Fix: replace |
||
|
|
||
| - config-keys: | ||
| - dsr1-fp8-b200-dynamo-trt | ||
| - dsr1-fp8-h200-dynamo-trt | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔴 The new
minimaxm2.5_fp4_b200.shscript usesEP_SIZEto conditionally enable expert parallelism (line 30) but omits it from thecheck_env_varsvalidation call (lines 5-13). IfEP_SIZEis not injected by the workflow, expert parallelism is silently disabled—producing incorrect, suboptimal benchmark results with no error. AddEP_SIZEtocheck_env_varsto match every other MiniMax benchmark script (fp8_b200, fp8_h100, fp8_h200, fp8_mi355x, fp8_mi325x).Extended reasoning...
What the bug is and how it manifests
The script
benchmarks/single_node/minimaxm2.5_fp4_b200.shreferencesEP_SIZEon line 30 via${EP_SIZE:-1}to decide whether to pass--enable-expert-parallelto the vLLM server. However,EP_SIZEis not listed in thecheck_env_varscall at lines 5–13. Thecheck_env_varsfunction is the repository's standard mechanism for early-failing when a required environment variable is missing—if it's not listed there, a missing value is never caught.The specific code path that triggers it
All six search-space entries in
nvidia-master.yamlforminimaxm2.5-fp4-b200-vllmspecifyep: 4orep: 2. The CI harness mapsep:to theEP_SIZEenvironment variable (viabenchmark-tmpl.ymlEP_SIZE: ${{ inputs.ep }}). If that injection is absent—due to a workflow misconfiguration, a manual execution, or a future refactor that omits the variable—the${EP_SIZE:-1}default silently evaluates to1, theifcondition is false, and--enable-expert-parallelis never passed to vLLM. The benchmark then runs in standard TP-only mode instead of the intended TP+EP mode.Why existing code doesn't prevent it
The
check_env_varsguard exists precisely to catch this class of silent misconfiguration. By omittingEP_SIZEfrom it, the script gives up the only defense against a missing value. Compare the otherwise identicalminimaxm2.5_fp8_b200.sh, which listsEP_SIZEas a required variable and uses$EP_SIZE(no default) in the same conditional—meaning a missing value there would both fail the pre-flight check and error at the[ $EP_SIZE -gt 1 ]comparison.Impact
MiniMax-M2.5 is a large MoE model where expert parallelism (EP) is a key performance lever. Running without EP when EP=4 or EP=2 is expected would significantly degrade benchmark throughput and latency, producing results that understate the hardware's actual capability. Because there is no error message or warning, these incorrect results would silently enter the benchmark record and potentially mislead performance comparisons.
How to fix it
Add
EP_SIZEto thecheck_env_varscall inminimaxm2.5_fp4_b200.sh, and consider removing the:-1default from${EP_SIZE:-1}so the script behaves identically to the FP8 variant—failing loudly whenEP_SIZEis absent rather than silently degrading.Step-by-step proof
EP_SIZEinjection):EP_SIZEis unset.check_env_varsis called withMODEL TP CONC ISL OSL MAX_MODEL_LEN RANDOM_RANGE_RATIO RESULT_FILENAME—it exits successfully becauseEP_SIZEis not listed.if [ "${EP_SIZE:-1}" -gt 1 ]→if [ "1" -gt 1 ]→ false.EPis set to" "(empty), so--enable-expert-parallelis never appended to the vLLM serve command.