Image attribution algorithms aim to identify important regions that arehighly relevant to model decisions. Although existing attribution solutions caneffectively assign importance to target elements, they still face the followingchallenges: 1) existing attribution methods generate inaccurate small regionsthus misleading the direction of correct attribution, and 2) the model cannotproduce good attribution results for samples with wrong predictions. To addressthe above challenges, this paper re-models the above image attribution problemas a submodular subset selection problem, aiming to enhance modelinterpretability using fewer regions. To address the lack of attention to localregions, we construct a novel submodular function to discover more accuratesmall interpretation regions. To enhance the attribution effect for allsamples, we also impose four different constraints on the selection ofsub-regions, i.e., confidence, effectiveness, consistency, and collaborationscores, to assess the importance of various subsets. Moreover, our theoreticalanalysis substantiates that the proposed function is in fact submodular.Extensive experiments show that the proposed method outperforms SOTA methods ontwo face datasets (Celeb-A and VGG-Face2) and one fine-grained dataset(CUB-200-2011). For correctly predicted samples, the proposed method improvesthe Deletion and Insertion scores with an average of 4.9% and 2.5% gainrelative to HSIC-Attribution. For incorrectly predicted samples, our methodachieves gains of 81.0% and 18.4% compared to the HSIC-Attribution algorithm inthe average highest confidence and Insertion score respectively. The code isreleased at https://github.com/RuoyuChen10/SMDL-Attribution.