Zhang Yi ; Li Dasong ; Shi Xiaoyu ; He Dailan ; Song Kangning ; Wang Xiaogang ; Qin Hongwei ; Li Hongsheng

Abstract

How to aggregate spatial information plays an essential role inlearning-based image restoration. Most existing CNN-based networks adopt staticconvolutional kernels to encode spatial information, which cannot aggregatespatial information adaptively. Recent transformer-based architectures achieveadaptive spatial aggregation. But they lack desirable inductive biases ofconvolutions and require heavy computational costs. In this paper, we propose akernel basis attention (KBA) module, which introduces learnable kernel bases tomodel representative image patterns for spatial information aggregation.Different kernel bases are trained to model different local structures. At eachspatial location, they are linearly and adaptively fused by predictedpixel-wise coefficients to obtain aggregation weights. Based on the KBA module,we further design a multi-axis feature fusion (MFF) block to encode and fusechannel-wise, spatial-invariant, and pixel-adaptive features for imagerestoration. Our model, named kernel basis network (KBNet), achievesstate-of-the-art performances on more than ten benchmarks over image denoising,deraining, and deblurring tasks while requiring less computational cost thanprevious SOTA methods.