One-sentence Summary

The authors propose pretrained audio neural networks (PANNs) trained on AudioSet, including a Wavegram-Logmel-CNN that combines log-mel spectrogram and waveform inputs, whose best system achieves a state-of-the-art mean average precision (mAP) of $0.439$0.439 on AudioSet tagging compared to the previous $0.392$0.392, transfers to six audio pattern recognition tasks, and is accompanied by released source code and pretrained models.

Key Contributions

• The paper introduces pretrained audio neural networks (PANNs) trained on AudioSet, which contains 1.9 million audio clips and 527 sound classes, and investigates the trade-off between tagging performance and computational complexity across multiple CNN architectures.
• It proposes Wavegram-Logmel-CNN, an architecture that combines waveform-derived Wavegram features with log-mel spectrogram input and achieves a state-of-the-art mAP of 0.439 on AudioSet tagging, compared with 0.392 for the previous best system and 0.314 for Google's system.
• Transfer experiments on six audio pattern recognition tasks show that PANNs outperform several prior state-of-the-art systems, and the source code and pretrained models are publicly released.

Introduction

Audio pattern recognition includes tasks such as audio tagging, acoustic scene classification, and sound event detection, but large-scale audio systems have been less developed than image or text models. A major obstacle is that AudioSet, the key large-scale audio dataset, was released as embedding features from a pretrained convolutional network rather than raw audio, which may limit downstream performance. Previous audio transfer learning also relied mainly on smaller music datasets. The authors introduce pretrained audio neural networks (PANNs) trained directly on raw AudioSet recordings and propose Wavegram-Logmel-CNN, which achieves a mean average precision of 0.439 on AudioSet tagging, outperforming the prior state-of-the-art and Google's system, while also transferring well to other audio pattern recognition tasks.

Dataset

The authors use AudioSet, a large-scale multi-label audio tagging dataset.

• Sources and composition:

  • AudioSet has an ontology of 527 sound classes.
  • Audio clips are extracted from YouTube videos.
  • The original training set has 2,063,839 clips, including a balanced subset of 22,160 clips with at least 50 clips per sound class.
  • The evaluation set has 20,371 clips.
  • Instead of using provided embedding features, the authors downloaded raw audio waveforms in December 2018 and discarded clips that were no longer downloadable.
  • Final downloaded data: 1,934,187 training clips, 20,550 balanced training clips, and 18,887 evaluation clips.

• Audio preprocessing:

  • Clips shorter than 10 seconds are padded with silence to 10 seconds.
  • All clips are converted to monophonic and resampled to 32 kHz.
  • For log mel spectrogram inputs, STFT uses a Hamming window of size 1024 and hop size 320, producing 100 frames per second.
  • 64 mel filter banks are used, with cutoff frequencies from 50 Hz to 14 kHz.
  • A 10-second audio clip becomes a log mel spectrogram of shape 1001 by 64.

• Class balancing:

  • AudioSet has a long-tailed distribution: frequent classes such as Speech and Music have over 900,000 clips, while rare classes such as Toothbrush have only tens of clips.
  • Training uses balanced sampling so mini-batches contain approximately equal numbers of clips from all sound classes.
  • The balance is approximate because an audio clip can contain multiple tags.

• Data augmentation:

  • Mixup is applied by default on log mel spectrograms, interpolating inputs and targets of two audio clips using a lambda sampled from a Beta distribution.
  • SpecAugment is applied on log mel spectrograms through frequency masking and time masking.

• Usage:

  • Models are trained on the downloaded AudioSet training clips and evaluated on the downloaded evaluation clips.
  • Balanced sampling prevents overfitting to frequent classes and underfitting to rare classes.
  • Mixup and SpecAugment help reduce overfitting, especially for sound classes with limited training clips.

Method

The authors leverage various convolutional neural network (CNN) architectures for audio tagging, primarily utilizing log mel spectrograms as input. These spectrograms are derived from time-domain waveforms via Short Time Fourier Transforms (STFTs) and mel filter banks. The base CNNs (6, 10, and 14 layers) utilize convolutional blocks with batch normalization and ReLU nonlinearity. Global pooling is applied after the final convolutional layer, combining maximum and average operations to summarize feature maps into fixed-length vectors. An extra fully-connected layer is added to enhance representation ability, followed by a linear classifier and sigmoid nonlinearity for tagging. The training minimizes a binary cross-entropy loss function:

where $f(x_{n})$f(xn​) represents the output probabilities for $K$K sound classes and $N$N is the number of training clips.

To address gradient propagation issues in deeper networks, the authors adapt Residual Networks (ResNets) with shortcut connections, implementing 22, 38, and 54-layer variants. For computational efficiency on portable devices, MobileNets are adapted using depthwise separable convolutions. Additionally, one-dimensional CNNs (1D CNNs) are explored to operate directly on time-domain waveforms, bypassing hand-crafted features. Variants include DaiNet, LeeNet, and a proposed Res1dNet with residual blocks and dilated convolutions to increase the receptive field.

Recognizing that 1D CNNs lack a frequency axis to capture pitch shifts, the authors propose the Wavegram, a learned time-frequency representation. To build a Wavegram, a 1D CNN is applied to the time-domain waveform. It starts with a convolutional layer (filter length 11, stride 5) followed by three convolutional blocks with dilations of 1 and 2, and downsampling layers. This reduces a 32 kHz recording to 100 frames per second. The output tensor $T \times C$T×C is reshaped to $T \times F \times C/F$T×F×C/F to introduce frequency bins, creating the Wavegram. This Wavegram serves as input to a 2D CNN backbone. Furthermore, the authors combine the Wavegram and log mel spectrogram along the channel dimension to create the Wavegram-Logmel-CNN system, utilizing information from both domains.

As shown in the figure below:

The architecture illustrates the parallel processing of the waveform into a Wavegram via 1D convolutional blocks and into feature maps via log mel spectrogram and 2D convolutional blocks. These representations are concatenated before passing through 2D CNN layers for prediction.

To handle the long-tailed distribution of AudioSet classes, a balanced sampling strategy is employed where audio clips are approximately equally sampled from all classes for each mini-batch. Data augmentation techniques include Mixup, which interpolates inputs and targets of two clips, and SpecAugment, which applies frequency and time masking to log mel spectrograms to improve robustness.

The authors investigate the generalization ability of Pre-trained Audio Neural Networks (PANNs) through transfer learning.

Refer to the framework diagram:

The figure outlines three strategies. First, a PANN is pretrained on the AudioSet dataset ($D_{AudioSet}$DAudioSet​). For a new task, the system can be trained from scratch as a baseline. Alternatively, the pretrained PANN can be used as a feature extractor where its parameters are frozen (indicated by the shaded rectangle), and a new classifier is built on the extracted embedding features. Finally, the PANN can be fine-tuned on the new task dataset ($D_{NewTask}$DNewTask​), where all parameters are initialized from the pretrained model except the final fully-connected layer, and all are updated during training.

Experiment

The experiments first evaluate PANNs on AudioSet tagging using mean average precision, AUC, and d-prime, testing architectural variants and training strategies such as data balancing, mixup augmentation, hop sizes, embedding dimensions, sample rates, and mel bins. These results show that deeper or learned-feature models improve tagging performance, with the Wavegram-Logmel-CNN achieving the best overall result, while MobileNets provide efficient lightweight alternatives. Transfer experiments on six audio tasks including acoustic scene classification, general audio tagging, music classification, and speech emotion recognition show that fine-tuned PANNs generalize well and consistently outperform models trained from scratch, especially when only limited training data are available.

The compared CNN architectures for AudioSet tagging use log-mel spectrogram inputs and differ mainly in temporal length, filter size, and depth. Pretrained PANNs transfer well to downstream audio classification tasks, with fine-tuned models generally outperforming models trained from scratch, though frozen features can be stronger when very few labeled clips are available. CNN14 and Wavegram-Logmel-CNN are among the strongest systems reported. CNN10 and CNN14 use stacks of 3x3 convolution blocks with batch normalization and ReLU, while CNN6 uses single 5x5 convolution layers. VGGish uses 96-frame log-mel spectrogram inputs, whereas CNN6, CNN10, and CNN14 use 1000-frame inputs. Frozen feature extractors perform better with very few training clips per class, but fine-tuned CNN14 surpasses them as more labeled clips become available. Fine-tuned PANNs achieve state-of-the-art or near state-of-the-art performance on several audio classification tasks and always outperform training from scratch.

This comparison of earlier AudioSet tagging methods shows that all learned models improve substantially over random guessing across mAP, AUC, and d-prime. Among the listed prior approaches, DeepRes reports the strongest metrics, with attention-based models and TAL Net close behind. Surrounding discussion notes that proposed PANNs later surpass these prior baselines, including CNN14 and ResNet38 outperforming the Google baseline by a notable margin. DeepRes leads the listed previous methods on mAP, AUC, and d-prime. Attention-based approaches and TAL Net trail DeepRes but remain far above the random guess baseline. Proposed PANNs in the wider results exceed these prior methods, with CNN14 and ResNet38 improving clearly over Google's baseline.

The fine-tuned model achieves the strongest result among the evaluated PANN training strategies, approaching the previous state-of-the-art and clearly outperforming the from-scratch model on DCASE 2018 Task 2. Freezing layers, especially the earliest layer, substantially reduces mAP@3, with Freeze_L3 performing only slightly better than Freeze_L1. On very small training sets, frozen-layer models can be beneficial, but fine-tuning becomes the strongest approach as more training clips are used. Fine-tuning leads the compared PANN variants on mAP@3 and nearly matches the state-of-the-art result. Freezing layers, particularly the first layer, causes a large performance drop compared with fine-tuning and training from scratch.

On RAVDESS emotion classification, the fine-tuned CNN14 achieves the highest reported accuracy, outperforming both training from scratch and the prior state-of-the-art system. Freezing the first or third layer of the pretrained network leads to a large performance drop, well below the other configurations. This suggests the RAVDESS audio distribution differs from AudioSet and that fine-tuning is necessary for strong transfer. Fine-tuned CNN14 yields the best RAVDESS accuracy among the compared systems, ahead of prior work and training from scratch. Freeze_L1 and Freeze_L3 obtain substantially lower accuracy, showing that using the pretrained audio network as a frozen feature extractor is ineffective for this task.

The experiments evaluate CNN architectures for AudioSet tagging with log-mel spectrogram inputs and then assess transfer learning across downstream audio classification tasks. Pretrained PANNs transfer well, with fine-tuned models generally outperforming training from scratch, while frozen feature extractors help mainly when very few labeled clips are available. Compared with earlier AudioSet methods, the proposed PANNs improve over strong prior baselines, and on DCASE 2018 Task 2 and RAVDESS emotion classification fine-tuning leads or approaches state-of-the-art, whereas freezing early layers causes substantial performance drops.