One-sentence Summary

The FAIR Team at Meta AI and The Hebrew University of Jerusalem propose AUDIOGEN, an auto-regressive text-to-audio generative model employing audio mixture augmentation to separate sources, multi-stream modeling for faster inference, and classifier-free guidance, achieving superior performance on both objective and subjective metrics across curated datasets.

Key Contributions

• AUDIOGEN, a Transformer-based auto-regressive model, learns discrete audio codes directly from raw waveforms and empirically demonstrates that text-conditioned auto-regressive models can generate high-quality audio, in contrast to prior diffusion-based approaches.
• An on-the-fly audio mixing augmentation randomly combines multiple audio samples during training, forcing the model to learn source separation and enabling generation of complex acoustic scenes from detailed captions.
• Multi-stream token processing reduces sequence length to accelerate inference while maintaining perceptual quality, and classifier-free guidance strengthens text alignment; objective and subjective evaluations show improved fidelity and adherence over baselines.

Introduction

Neural generative models have rapidly advanced image synthesis, but applying similar textual guidance to high-fidelity audio remains challenging. Audio is a one-dimensional signal with overlapping sounds, reverberation, and distinct psychoacoustic properties, and paired text-audio data is far scarcer than image-text data. Previous text-to-audio approaches like DiffSound rely on mel-spectrogram-based discrete codes and diffusion processes, leaving autoregressive transformer models under-explored. The authors introduce AUDIOGEN, a two-stage model that first learns discrete audio tokens directly from raw waveforms via neural compression, then trains an autoregressive transformer-decoder conditioned on pretrained T5 text embeddings. By incorporating on-the-fly text and audio mixing augmentation and classifier-free guidance, AUDIOGEN generates diverse, controllable audio compositions from descriptive captions and outperforms prior work in both objective and subjective evaluations, demonstrating that autoregressive models can achieve state-of-the-art audio generation.

Method

The method operates in two primary stages: first, learning a discrete representation of raw audio using an auto-encoding approach, and second, training a Transformer language model over the learned audio codes conditioned on textual features. During inference, the system samples from the language model to generate audio tokens given text features, which are subsequently decoded back into the waveform domain.

Audio Representation The audio representation model consists of an encoder network $E$E, a quantization layer $Q$Q, and a decoder network $G$G. The encoder processes an audio segment to output a latent representation $z$z. The quantization layer, utilizing a Residual Vector Quantization module, produces a compressed representation $z_q$zq​. The decoder then reconstructs the time-domain signal $\hat{x}$x^ from $z_q$zq​. The encoder architecture comprises a 1D convolution followed by convolutional blocks, each containing a residual unit and a strided convolution for down-sampling. These blocks are followed by a two-layer LSTM and a final 1D convolution. The decoder mirrors this structure using transposed convolutions.

The system is trained end-to-end to minimize a combination of reconstruction and adversarial losses. The time-domain loss minimizes the L1 distance between the target and reconstructed audio: $\ell_t(x, \hat{x}) = \|x - \hat{x}\|_1$ℓt​(x,x^)=∥x−x^∥1​. The frequency domain loss uses a linear combination of L1 and L2 losses over multi-scale mel-spectrograms: $\ell_f(x, \hat{x}) = \frac{1}{|\alpha| \cdot |s|} \sum_{\alpha_i \in \alpha} \sum_{i \in e} \| \mathcal{S}_i(x) - \mathcal{S}_i(\hat{x}) \|_1 + \alpha_i \| \mathcal{S}_i(x) - \mathcal{S}_i(\hat{x}) \|_2$ℓf​(x,x^)=∣α∣⋅∣s∣1​∑αi​∈α​∑i∈e​∥Si​(x)−Si​(x^)∥1​+αi​∥Si​(x)−Si​(x^)∥2​ To enhance generation quality, the authors also optimize a multi-scale STFT-based discriminator and include a feature matching loss for the generator: $\ell_{feat}(x, \hat{x}) = \frac{1}{KL} \sum_{k=1}^K \sum_{l=1}^L \| D_k^l(x) - D_k^l(\hat{x}) \|_1$ℓfeat​(x,x^)=KL1​∑k=1K​∑l=1L​∥Dkl​(x)−Dkl​(x^)∥1​ The total generator loss is formulated as $\mathcal{L}_G = \lambda_t \ell_t(x, \hat{x}) + \lambda_f \ell_f(x, \hat{x}) + \lambda_g \ell_g(\hat{x}) + \lambda_{feat} \ell_{feat}(x, \hat{x})$LG​=λt​ℓt​(x,x^)+λf​ℓf​(x,x^)+λg​ℓg​(x^)+λfeat​ℓfeat​(x,x^).

Audio Language Modeling To generate audio conditioned on text, the Audio Language Model outputs a sequence of audio tokens $\hat{z}_q$z^q​. A pre-trained text encoder maps the raw text input $c$c into a semantic dense representation $u$u. Simultaneously, a Look-Up-Table embeds the audio tokens into a continuous space $v$v. The text and audio embeddings are concatenated to form the input sequence.

The authors train a Transformer-decoder language model parameterized by $\theta$θ using the cross-entropy loss: $L_{\text{LM}} = - \sum_{i=1}^N \sum_{j=1}^{T_v} \log p_\theta(v_j^i | u_1^i, \dots, u_{T_u}^i, v_1^i, \dots, v_{j-1}^i)$LLM​=−∑i=1N​∑j=1Tv​​logpθ​(vji​∣u1i​,…,uTu​i​,v1i​,…,vj−1i​) To improve text adherence, cross-attention between audio and text is added to each attention block of the transformer.

The authors also implement Classifier-Free Guidance to control the trade-off between sample quality and diversity. During training, text conditioning is randomly omitted in a portion of samples. At inference, sampling is performed from a linear combination of conditional and unconditional probabilities: $\gamma \log p_\theta(v_j^i | u_1^i, \dots, u_{T_u}^i, v_1^i, \dots, v_{j-1}^i) + (1 - \gamma) \log p_\theta(v_j^i | v_1^i, \dots, v_{j-1}^i)$γlogpθ​(vji​∣u1i​,…,uTu​i​,v1i​,…,vj−1i​)+(1−γ)logpθ​(vji​∣v1i​,…,vj−1i​) where $\gamma$γ is the guidance scale.

To handle the long sequences required for high-quality audio, the authors propose a Multi-Stream representation. By generalizing the quantization to a Residual Vector Quantization module with $k$k streams, the sequence length is reduced. At each time step, the network processes $k$k discrete codes embedded via $k$k embedding layers, with the final embedding being the mean of these $k$k embeddings. The network utilizes $k$k independent language model prediction heads to output the $k$k codes.

Experiment

The evaluation setup combines diverse audio datasets with a mixing augmentation that creates novel audio-text compositions, and both objective and subjective metrics are used to assess AUDIOGEN models against DiffSound. AUDIOGEN consistently outperforms the baseline, with the mixing strategy notably improving relevance to complex text prompts. Audio continuation experiments show that textual guidance is strongest with short prompts, while longer prompts rely more on audio context. Ablation studies indicate a classifier-free guidance scale of 3.0 offers the best quality-diversity trade-off, and multi-stream representations accelerate inference at the expense of generation quality.

AUDIOGEN-base with 285M parameters outperforms the larger DiffSound (400M) across all reported subjective and objective metrics. Adding mixing augmentations improves text relevance scores (lower KL, higher REL) while overall quality remains comparable, at a slight cost in FAD. The 1B-parameter large model achieves the best results, and even when trained on the same AudioSet and AudioCaps data, AUDIOGEN-base reduces FAD from 7.39 to 4.39 and KL from 2.57 to 2.46 relative to DiffSound. A smaller AUDIOGEN-base model surpasses DiffSound on every measured metric, including subjective overall quality and text relevance. Mixing augmentation boosts text relevance (KL drops from 2.14 to 2.09, REL rises from 63.23 to 66.01) at the cost of a small FAD increase, while overall quality stays similar. AUDIOGEN-large with 1B parameters and T5-large text conditioning sets the lowest FAD (1.82) and KL (1.69) and the highest subjective relevance (68.73). When both are trained on AudioSet plus AudioCaps, AUDIOGEN-base cuts FAD nearly in half (4.39 vs 7.39) and obtains a better KL score (2.46 vs 2.57) than DiffSound.

Multi-stream audio representations preserve perceptual fidelity as measured by ViSQOL despite reduced bitrate and SI-SNR. Generation quality, measured by FAD and KL, degrades considerably with additional streams while inference speed increases substantially. This establishes a direct trade-off between generation quality and inference speed. All multi-stream configurations yield ViSQOL scores close to 3.9, comparable to the single-stream model, even as SI-SNR drops from 5.1 dB (single) to 4.2 dB (four streams). For the large model, moving from one stream to four streams increases FAD from 1.82 to 10.89 and KL from 1.69 to 2.59, but provides a 3.6× inference speed-up.

The experiments evaluate audio generation quality using FAD, KL, and subjective ratings, first showing that AudioGen models consistently outperform DiffSound, where even a smaller base model surpasses the larger baseline, and mixing augmentations boost text relevance with only a minor fidelity cost while scaling to 1 billion parameters yields the best overall performance. A separate multi-stream analysis demonstrates that perceptual fidelity remains high across configurations, but generation metrics worsen markedly as stream count rises, and inference speed increases substantially, revealing a direct trade-off between generation quality and speed.