Reviewer handbook > Area-specific reviewing instructions
We (IS 2025 TPC) are starting a new initiative to collect area-specific reviewing guidelines. These are optional, but can help reviewers to pay attention to specific aspects of the papers in their areas.
¶ Area 7: Speech Synthesis and Spoken Language Generation
In January 2025, several researchers with a background in speech and speech synthesis evaluation came together for a Dagstuhl seminar titled ”Task and Situation-Aware Evaluation of Speech and Speech Synthesis” organized by Jens Edlund, Sébastien Le Maguer, Christina Tånnander and Petra Wagner. Many Speech Synthesis papers do not adequately describe the details of the subjective evaluation experiments. In order help reviewers of such papers, they came up with a checklist of good practices for evaluation of speech synthesis. For people interested in reading the whole document, please follow the link here. The shorter checklist is listed below:
- The chosen evaluation protocol(s) should address and verify the claims being made.
- The synthesis use case addressed in the paper should be incorporated in the evaluation.
- Papers targeting ``no use case'' synthesis with basic MOS evaluations should especially take care to ensure that their evaluation protocol can reveal meaningful distinctions between systems, and should especially include some analysis about why their system outperforms baselines.
- Papers should not necessarily be penalized for not including a MOS test. Reviewer recommendations to include a MOS test should also include recommendations for relevant comparison systems.
- ``Naturalness'' is under-specified and authors should ensure that they share an understanding with listeners about what to evaluate.
- Existing protocols should be referenced properly, and new protocols should be described in sufficient detail. Alterations to existing protocols should be justified and described in sufficient detail.
- Descriptions of listening tests should include the following minimum necessary information: how many listeners participated, how many audio samples per system, how many ratings per sample, the type of statistical test performed, the question asked to the listeners, and the rating scale and its increments if one was used.
- Listeners should be qualified to evaluate the hypothesis, and the listener population should be sufficiently described.
- Enough listeners should be recruited for the listening test to have sufficient statistical power.
- An appropriate statistical test should be conducted.
- Baseline comparison systems should be sufficiently described for reproducibility, and sensibly-chosen to evaluate the claims of the paper.
- The test material chosen should be described and sensibly-chosen to evaluate the claims made by the paper.
- MOS values cannot be meaningfully compared across different papers or across separate listening tests.
- Use and choice of automatic quality predictors should be clearly stated, well-justified, and sufficiently described.