Inquiry regarding reproduction of "Emergent TTS Eval" results

[hujingbin1]

Self Checks

• This template is only for bug reports. For questions, please visit Discussions.
• I have thoroughly reviewed the project documentation (installation, training, inference) but couldn't find information to solve my problem. English 中文 日本語 Portuguese (Brazil)
• I have searched for existing issues, including closed ones. Search issues
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template and fill in all required fields.

Cloud or Self Hosted

Self Hosted (Source)

Environment Details

Python 3.10

Steps to Reproduce

I would like to extend my sincere gratitude to the team for fully open-sourcing the Fish Audio S2 model, including the weights, fine-tuning code, and inference engine. This is a significant contribution to the open-source community and has greatly lowered the barrier to entry for high-quality speech synthesis, allowing us to learn from and utilize such advanced technology.

However, I am currently attempting to reproduce the results reported in the Fish Audio S2 technical report on the Emergent TTS Bench, but I am facing a significant performance gap. My current reproduction yields an Overall Win Rate of 0.2534, which is far lower than the results reported in the paper.

For my current test setup, I utilized the raw text from the evaluation set and did not use prompt audio as a condition. To help identify the discrepancy, could you please clarify the following details regarding your experimental setup?
Text Processing: What specific text processing pipeline was applied to the evaluation data?
Model Selection: Were the scores achieved using the public open-source weights or an internal model?
Inference Mode: Was the evaluation conducted using voice cloning (with prompt conditions) or unconditional generation?

Thanks!

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

[PoTaTo-Mika]

All produced by the open-source model.
We first generate the audio into files, then evaluated the metrics offline with Gemini.
You may check your reference audio, this will strongly affect the final win-rate. A piece of native english reference audio will increase your win-rate against other models.
@Whale-Dolphin helped me with this benchmark in the further tests.

And for your questions:

1. Text frontend is extract from our online service (not open-sourced but you can use LLM instead)
2. S2-Pro
3. Voice cloning with several reference audio, the best one's performance is written in the paper.

[Whale-Dolphin]

In our subjective evaluation pipeline, we found that the reference audio has a huge impact on the results. On Common Voice (using a subset from Seed-TTS-Eval), we selected a set of high-quality audio prompts for inference. Prompt audio should always be used during inference, as omitting it leads to a substantial degradation in model performance.

In addition, we used the same normalization frontend as our online serve, since we believe that TTS without a text frontend is of very limited practical value. No serious TTS provider would generate speech without first applying a text normalization (TN) frontend.

What’s more, we used Gemini 3 Pro to rewrite the text with instruction tags, which can significantly improve performance in certain domains.

FYI