Ding Yiwei ; Lerch Alexander

Abstract

Music classification has been one of the most popular tasks in the field ofmusic information retrieval. With the development of deep learning models, thelast decade has seen impressive improvements in a wide range of classificationtasks. However, the increasing model complexity makes both training andinference computationally expensive. In this paper, we integrate the ideas oftransfer learning and feature-based knowledge distillation and systematicallyinvestigate using pre-trained audio embeddings as teachers to guide thetraining of low-complexity student networks. By regularizing the feature spaceof the student networks with the pre-trained embeddings, the knowledge in theteacher embeddings can be transferred to the students. We use variouspre-trained audio embeddings and test the effectiveness of the method on thetasks of musical instrument classification and music auto-tagging. Results showthat our method significantly improves the results in comparison to theidentical model trained without the teacher's knowledge. This technique canalso be combined with classical knowledge distillation approaches to furtherimprove the model's performance.