Abstract

In this report, we introduce Qwen2.5, a comprehensive series of largelanguage models (LLMs) designed to meet diverse needs. Compared to previousiterations, Qwen 2.5 has been significantly improved during both thepre-training and post-training stages. In terms of pre-training, we have scaledthe high-quality pre-training datasets from the previous 7 trillion tokens to18 trillion tokens. This provides a strong foundation for common sense, expertknowledge, and reasoning capabilities. In terms of post-training, we implementintricate supervised finetuning with over 1 million samples, as well asmultistage reinforcement learning. Post-training techniques enhance humanpreference, and notably improve long text generation, structural data analysis,and instruction following. To handle diverse and varied use cases effectively,we present Qwen2.5 LLM series in rich sizes. Open-weight offerings include baseand instruction-tuned models, with quantized versions available. In addition,for hosted solutions, the proprietary models currently include twomixture-of-experts (MoE) variants: Qwen2.5-Turbo and Qwen2.5-Plus, bothavailable from Alibaba Cloud Model Studio. Qwen2.5 has demonstrated top-tierperformance on a wide range of benchmarks evaluating language understanding,reasoning, mathematics, coding, human preference alignment, etc. Specifically,the open-weight flagship Qwen2.5-72B-Instruct outperforms a number of open andproprietary models and demonstrates competitive performance to thestate-of-the-art open-weight model, Llama-3-405B-Instruct, which is around 5times larger. Qwen2.5-Turbo and Qwen2.5-Plus offer superior cost-effectivenesswhile performing competitively against GPT-4o-mini and GPT-4o respectively.Additionally, as the foundation, Qwen2.5 models have been instrumental intraining specialized models such as Qwen2.5-Math, Qwen2.5-Coder, QwQ, andmultimodal models.