Abstract

The salient multimodal capabilities and interactive experience of GPT-4ohighlight its critical role in practical applications, yet it lacks ahigh-performing open-source counterpart. In this paper, we introduceBaichuan-Omni, the first open-source 7B Multimodal Large Language Model (MLLM)adept at concurrently processing and analyzing modalities of image, video,audio, and text, while delivering an advanced multimodal interactive experienceand strong performance. We propose an effective multimodal training schemastarting with 7B model and proceeding through two stages of multimodalalignment and multitask fine-tuning across audio, image, video, and text modal.This approach equips the language model with the ability to handle visual andaudio data effectively. Demonstrating strong performance across variousomni-modal and multimodal benchmarks, we aim for this contribution to serve asa competitive baseline for the open-source community in advancing multimodalunderstanding and real-time interaction.