Maxim Zhelnin Viktor Moskvoretskii Egor Shvetsov Egor Venediktov Mariya Krylova Aleksandr Zuev Evgeny Burnaev

Abstract

Parameter Efficient Fine-Tuning (PEFT) methods have gained popularity anddemocratized the usage of Large Language Models (LLMs). Recent studies haveshown that a small subset of weights significantly impacts performance. Basedon this observation, we introduce a novel PEFT method, called Gaussian noiseInjected Fine Tuning of Salient Weights (GIFT-SW). Our method updates onlysalient columns, while injecting Gaussian noise into non-salient ones. Toidentify these columns, we developeda generalized sensitivity metric thatextends and unifies metrics from previous studies. Experiments with LLaMAmodels demonstrate that GIFT-SW outperforms full fine-tuning and modern PEFTmethods under the same computational budget. Moreover, GIFT-SW offers practicaladvantages to recover performance of models subjected to mixed-precisionquantization with keeping salient weights in full precision.