Multi-LMentry is a multilingual benchmark dataset released in 2025, designed to systematically evaluate the cross-lingual generalization ability of large language models (LLMs) for low-level language understanding and basic reasoning tasks in multilingual environments. This dataset covers nine languages: English, Catalan, German, Spanish, Basque, Galician, Korean, Italian, and Brazilian Portuguese. The tasks were manually redesigned by native speakers, similar in form to the original LMentry framework, but not as direct translations, to ensure naturalness and cultural fit.

Dataset structure

• The dataset is organized into folders by language.
• In each language folder, each task corresponds to a JSON file.
• Each JSON file contains input hints and expected outputs for the task.
• The task types include simple sentence construction, contextual vocabulary selection, and letter reasoning.
• Some tasks are language-specific; for example, rhyming tasks are excluded in languages where they are not applicable.

Citation

@inproceedings{moroni-etal-2025-multi,
title = "Multi-{LM}entry: Can Multilingual {LLM}s Solve Elementary Tasks Across Languages?",
author = "Moroni, Luca  and
Aula-Blasco, Javier  and
Conia, Simone  and
Baucells, Irene  and
Perez, Naiara  and
Su{\'a}rez, Silvia Paniagua  and
Sall{\'e}s, Anna  and
Ostendorff, Malte  and
Falc{\~a}o, J{\'u}lia  and
Son, Guijin  and
Gonzalez-Agirre, Aitor  and
Navigli, Roberto  and
Villegas, Marta",
editor = "Christodoulopoulos, Christos  and
Chakraborty, Tanmoy  and
Rose, Carolyn  and
Peng, Violet",
booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2025",
address = "Suzhou, China",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.emnlp-main.1731/",
doi = "10.18653/v1/2025.emnlp-main.1731",
pages = "34114--34145",
ISBN = "979-8-89176-332-6"
}