Sangmin Bae Bilge Acun Haroun Habeeb Seungyeon Kim Chien-Yu Lin Liang Luo Junjie Wang Carole-Jean Wu

Abstract

Recent progress in large language models demonstrates that hybridarchitectures--combining self-attention mechanisms with structured state spacemodels like Mamba--can achieve a compelling balance between modeling qualityand computational efficiency, particularly for long-context tasks. While thesehybrid models show promising performance, systematic comparisons ofhybridization strategies and analyses on the key factors behind theireffectiveness have not been clearly shared to the community. In this work, wepresent a holistic evaluation of hybrid architectures based on inter-layer(sequential) or intra-layer (parallel) fusion. We evaluate these designs from avariety of perspectives: language modeling performance, long-contextcapabilities, scaling analysis, and training and inference efficiency. Byinvestigating the core characteristics of their computational primitive, weidentify the most critical elements for each hybridization strategy and furtherpropose optimal design recipes for both hybrid models. Our comprehensiveanalysis provides practical guidance and valuable insights for developinghybrid language models, facilitating the optimization of architecturalconfigurations.