One-sentence Summary

Researchers from Sapienza University of Rome, Apple, and Adobe introduce XC-Translate, the first large-scale benchmark for cross-cultural machine translation targeting culturally-nuanced entity names, and KG-MT, which integrates multilingual knowledge graphs into neural machine translation via dense retrieval, achieving 129% and 62% relative improvements over NLLB-200 and GPT-4, respectively.

Key Contributions

• The paper introduces XC-Translate, the first large-scale, manually created benchmark for machine translation of text containing culturally nuanced entity names, designed to test cross-cultural adaptation beyond transliteration.
• The paper presents KG-MT, an end-to-end neural machine translation method that retrieves relevant entity names from a multilingual knowledge graph through dense retrieval and integrates them into translation instead of relying on synthetic augmentation of every possible entity translation.
• Experiments on XC-Translate and existing machine translation benchmarks show that KG-MT achieves 129% and 62% relative improvements over NLLB-200 and GPT-4, respectively, in translating texts containing entity names, while maintaining comparable results on general-purpose machine translation benchmarks.

Introduction

Multilingual large language models and massive multilingual datasets have greatly improved machine translation, especially for high-resource languages. However, translating text that contains entity names, such as movies, books, people, or locations, remains challenging because literal translations may fail to capture culturally specific references and can produce semantically incorrect results.

Prior work has limited progress in this area. Earlier methods often focus on transliteration rather than cross-cultural name translation, especially for languages that share the same script, and large-scale high-quality benchmarks are missing. Approaches based on synthetic data augmentation can help, but they increase training costs and require frequent updates as new entities appear.

The authors address this gap with two main contributions. They introduce XC-Translate, a large-scale manually created benchmark for cross-cultural translation of text containing entity names across 10 language pairs. They also propose KG-MT, a machine translation system that uses retrieval-augmented generation from multilingual knowledge graphs to retrieve relevant entity name translations on the fly, improving performance while requiring less supervision and fewer computational resources than data augmentation methods.

Dataset

• Dataset: Cross-Culture Translate (XC-Translate), a manually curated benchmark for cross-cultural machine translation of texts that contain entity names.

• Language pairs: 10 English-to-X pairs covering diverse high- and low-resource languages: Arabic, Chinese, French, German, Italian, Japanese, Korean, Spanish, Thai, and Turkish.

• Scale: About 5,000 source sentences for each language pair, totaling over 58,000 instances.

• Reference set: Multiple gold translations are provided for each source text, with over 100,000 references in total, or roughly 2 references per sentence on average.

• Entity sourcing: Entities are selected from Wikidata for each language pair using two filters:

  • The entity must have at least one name in English and one name in the target language.
  • The English name must differ by at least 50% from the target-language name compared to word-for-word translations into English, measured using Levenshtein distance.
  • This helps select entities whose names are likely to involve cross-cultural translation challenges rather than simple transliteration.

• Source sentence construction: Human annotators create short English knowledge-seeking questions, each under 25 words, about the selected entities. The question format is used to keep texts simple and to reduce the risk of introducing incorrect factual statements.

• Translation pipeline: Each English question is translated into the target language through a two-step human process:

  • At least 3 translators produce candidate translations.
  • At least 3 annotators verify each translation.
  • Only translations agreed upon by annotators are retained.

• Annotator controls: Translators and verifiers must be fluent in English, native speakers of the target language, and residents in countries where the target language is spoken. They must also pass an entrance test and periodic quality checks.

• Context provided during translation: Translators receive entity names, entity descriptions from Wikidata, and Wikipedia pages for the entity in both English and the target language to support correct name translation and contextual understanding.

• Dataset usage: The paper uses XC-Translate for evaluation of cross-cultural translation methods, especially for assessing how well models translate entity names across different languages and scripts. No training split or mixture ratio is described for this benchmark.

Method

The authors leverage a multilingual knowledge graph to move machine translation from entity memorization to entity-aware retrieval and generation. Instead of augmenting training data with synthetic transliterations for every possible entity, the proposed framework first retrieves the most relevant entities for a source text and then conditions the translator on their names in the target language. The framework diagram provides an overview of the overall pipeline.

Given a source text $\mathbf{t} = \langle w_1, \ldots, w_n \rangle$t=⟨w1​,…,wn​⟩ in a source language $l_s$ls​, the knowledge retriever selects the top-$k$k relevant entities from a knowledge graph $G = \mathcal{E} \times \mathcal{R} \times \mathcal{E}$G=E×R×E, where $\mathcal{E}$E is the set of entities and $\mathcal{R}$R is the set of relations. Each entity $e_i$ei​ is represented as a tuple $e_i = \langle n_i, d_i \rangle$ei​=⟨ni​,di​⟩, where $n_i$ni​ is the primary entity name and $d_i$di​ is its description. Including the description helps distinguish homonymous entities that share the same name.

The retriever assigns a relevance score between an entity embedding $\mathbf{e}_i$ei​ and the source text embedding $\mathbf{t}$t using cosine similarity:

The top-$k$k entity set is then obtained as:

The entity encoder is trained with a contrastive objective that encourages retrieval of relevant entities and discourages retrieval of irrelevant entities. For a relevant positive entity $e^+$e+ and irrelevant negative entities $e^-$e−, the training loss is:

A key design detail is the hard-negative sampling strategy. Instead of sampling negative entities randomly from the batch, the method selects homonymous entities that have the same name as the relevant entity but are not relevant to the source text. This pushes the retriever to rely on contextual disambiguation rather than surface string matching.

The knowledge-enhanced translator receives the source text together with retrieved entity mappings. The authors construct a knowledge-augmented source sequence:

where $[\mathrm{KG}]$[KG] is a special marker for the start of entity information, $n_i^s$nis​ is the entity name in the source language, and $n_i^t$nit​ is the corresponding name in the target language retrieved from the multilingual knowledge graph. This augmented sequence is passed to a standard sequence-to-sequence machine translation model, which is fine-tuned to generate the target text $\mathbf{t}' = \langle w_1', \ldots, w_m' \rangle$t′=⟨w1′​,…,wm′​⟩ while attending to the target-language entity names. The authors refer to this mechanism as explicit knowledge integration.

The framework also supports implicit knowledge integration by using the latent representations learned by the retriever. After encoding the augmented source sequence $\mathbf{t}^{+\mathrm{KG}}$t+KG with the MT encoder, the model obtains hidden states $\mathbf{h}^{+\mathrm{KG}} = \langle \mathbf{h}_1, \ldots, \mathbf{h}_{n+k+1} \rangle$h+KG=⟨h1​,…,hn+k+1​⟩. The entity embeddings $\mathbf{e}_i$ei​ from the retriever are then prepended to these hidden states:

The decoder then attends to this combined representation. This fusion allows the translator to use both the explicit surface forms from the knowledge graph and the finer-grained semantic information stored in the retriever’s entity embeddings. The architecture fuses hidden states from two different encoders, namely the knowledge retriever and the machine translation encoder, rather than operating as a conventional single-encoder Fusion-in-Decoder system.

Experiment

Experiments assess entity-name translation in XC-Translate using BLEU, COMET, and the proposed entity-level M-ETA, while also checking general translation quality on WMT benchmarks. Results show that conventional MT systems and LLMs struggle with entity names, whereas KG-MT consistently improves entity accuracy and remains competitive on standard translation tasks. Ablations further confirm that knowledge retrieval is effective and that both explicit and implicit knowledge integration help, identifying knowledge integration as the main remaining challenge.

The comparison averages results on XC-Translate across BLEU, COMET, and M-ETA, contrasting large language models with compact multilingual MT systems. Larger LLMs, especially GPT-4, show stronger overall translation quality than the compact MT baselines, while vanilla MT systems have weaker entity-sensitive performance. The surrounding analysis indicates that KG-MT benefits from both explicit and implicit knowledge integration, with their combination performing best. GPT-4 leads the listed systems on the reported translation quality metrics. NLLB-200 is the strongest compact multilingual MT baseline, but still trails the strongest LLM. Lower M-ETA scores for vanilla MT systems suggest unreliable handling of knowledge-dependent translation. Combining explicit and implicit knowledge integration improves KG-MT more than either method alone. Using gold entities only modestly improves KG-MT, pointing to knowledge integration as a bottleneck.

The results compare multilingual MT systems and LLMs across language pairs using BLEU and entity-aware M-ETA, giving the baseline context for KG-MT. Among the reported systems, GPT-4 generally performs best, especially on English-to-European pairs. The shared weakness on English-to-Thai, and on many pairs for entity-aware scoring, shows that vanilla systems often struggle to handle knowledge-sensitive translation. GPT-4 leads the listed systems on most language pairs in both BLEU and entity-aware M-ETA. Open multilingual models such as NLLB-200 are competitive, but they remain below the strongest LLM on most pairs. English-to-Thai scores are very low across systems, indicating a major difficulty for entity-aware translation quality.

Across the evaluated English-to-other-language settings, larger and newer large language models achieve stronger translation quality than the smaller model, with GPT-4 performing best overall. The compared multilingual machine translation models are also competitive, with NLLB-200 scoring closest to the strongest LLM while improving over the other multilingual systems. GPT-4 outperforms both GPT-3.5 and GPT-3 on the averaged translation metrics. NLLB-200 is the strongest compared multilingual machine translation model and is close to GPT-4.

The experiments evaluate large language models and compact multilingual machine translation systems on multilingual translation and entity-sensitive settings, while also examining knowledge integration in KG-MT. Larger LLMs, especially GPT-4, generally provide stronger overall translation quality than compact baselines, with NLLB-200 as the strongest compact model but still trailing the leading LLM. Vanilla MT systems show weaker knowledge-dependent performance, notably on difficult pairs such as English-to-Thai, and KG-MT improves most when explicit and implicit knowledge are combined, indicating that effective knowledge integration is a key bottleneck.