Enze Zhang Jiaying Wang Mengxi Xiao Jifei Liu Ziyan Kuang Rui Dong Eric Dong Sophia Ananiadou Min Peng Qianqian Xie

Abstract

Large language models (LLMs) have substantially advanced machine translation(MT), yet their effectiveness in translating web novels remains unclear.Existing benchmarks rely on surface-level metrics that fail to capture thedistinctive traits of this genre. To address these gaps, we introduce DITING,the first comprehensive evaluation framework for web novel translation,assessing narrative and cultural fidelity across six dimensions: idiomtranslation, lexical ambiguity, terminology localization, tense consistency,zero-pronoun resolution, and cultural safety, supported by over 18Kexpert-annotated Chinese-English sentence pairs. We further propose AgentEval,a reasoning-driven multi-agent evaluation framework that simulates expertdeliberation to assess translation quality beyond lexical overlap, achievingthe highest correlation with human judgments among seven tested automaticmetrics. To enable metric comparison, we develop MetricAlign, a meta-evaluationdataset of 300 sentence pairs annotated with error labels and scalar qualityscores. Comprehensive evaluation of fourteen open, closed, and commercialmodels reveals that Chinese-trained LLMs surpass larger foreign counterparts,and that DeepSeek-V3 delivers the most faithful and stylistically coherenttranslations. Our work establishes a new paradigm for exploring LLM-based webnovel translation and provides public resources to advance future research.