Xilin Wei Xiaoran Liu Yuhang Zang Xiaoyi Dong Yuhang Cao Jiaqi Wang Xipeng Qiu Dahua Lin

Abstract

Implicit Chain-of-Thought (CoT) methods present a promising, token-efficientalternative to explicit CoT reasoning in Large Language Models (LLMs), but apersistent performance gap has limited the application of implicit CoT. Weidentify a core latent instability issue by scaling the computational budget ofimplicit CoT approaches: as we increase the number of implicit reasoning tokensto enhance performance, the training process often becomes unstable andcollapses. Our analysis reveals that this instability arises from the latentrepresentations becoming homogeneous and losing their semantic diversity, afailure caused by insufficient step-level supervision in existing implicit CoTapproaches. To address this issue, we propose SIM-CoT, a plug-and-play trainingmodule that introduces step-level supervision to stabilize and enrich thelatent reasoning space. Specifically, SIM-CoT employs an auxiliary decoderduring training to align each implicit token with its corresponding explicitreasoning step, ensuring that latent states capture distinct and meaningfulinformation. The proposed auxiliary decoder is removed during inference,preserving the computational efficiency of implicit CoT methods with no addedoverhead. In addition, the auxiliary decoder affords interpretability ofimplicit reasoning by projecting each latent token onto an explicit reasoningvocabulary, enabling per-step visualization of semantic roles and diagnosis.SIM-CoT significantly enhances both the in-domain accuracy and out-of-domainstability of various implicit CoT methods, boosting baselines like Coconut by+8.2% on GPT-2 and CODI by +3.0% on LLaMA-3.1 8B. Demonstrating strongscalability, SIM-CoT also surpasses the explicit CoT baseline on GPT-2 by 2.1%with 2.3\times greater token efficiency, while substantially closing theperformance gap on larger models like LLaMA-3.1 8B.