Kong Zhifeng ; Goel Arushi ; Badlani Rohan ; Ping Wei ; Valle Rafael ; Catanzaro Bryan

Abstract

Augmenting large language models (LLMs) to understand audio -- includingnon-speech sounds and non-verbal speech -- is critically important for diversereal-world applications of LLMs. In this paper, we propose Audio Flamingo, anovel audio language model with 1) strong audio understanding abilities, 2) theability to quickly adapt to unseen tasks via in-context learning and retrieval,and 3) strong multi-turn dialogue abilities. We introduce a series of trainingtechniques, architecture design, and data strategies to enhance our model withthese abilities. Extensive evaluations across various audio understanding tasksconfirm the efficacy of our method, setting new state-of-the-art benchmarks.Our demo website is https://audioflamingo.github.io/ and the code isopen-sourced at https://github.com/NVIDIA/audio-flamingo.