Chen Shengzhuang ; Tack Jihoon ; Yang Yunqiao ; Teh Yee Whye ; Schwarz Jonathan Richard ; Wei Ying

Abstract

Recent successes suggest that parameter-efficient fine-tuning of foundationmodels as the state-of-the-art method for transfer learning in vision,replacing the rich literature of alternatives such as meta-learning. In tryingto harness the best of both worlds, meta-tuning introduces a subsequentoptimization stage of foundation models but has so far only shown limitedsuccess and crucially tends to underperform on out-of-distribution (OOD) tasks.In this paper, we introduce Sparse MetA-Tuning (SMAT), a method inspired bysparse mixture-of-experts approaches and trained to isolate subsets ofpre-trained parameters automatically for meta-tuning on each task. SMATsuccessfully overcomes OOD sensitivity and delivers on the promise of enhancingthe transfer abilities of vision foundation models beyond parameter-efficientfine-tuning. We establish new state-of-the-art results on a challengingcombination of Meta-Dataset augmented with additional OOD tasks in bothzero-shot and gradient-based adaptation settings. In addition, we provide athorough analysis of the superiority of learned over hand-designed sparsitypatterns for sparse expert methods and the pivotal importance of the sparsitylevel in balancing between in-distribution and out-of-distributiongeneralization. Our code is publicly available.