Qi Zekun ; Dong Runpei ; Fan Guofan ; Ge Zheng ; Zhang Xiangyu ; Ma Kaisheng ; Yi Li

Abstract

Mainstream 3D representation learning approaches are built upon contrastiveor generative modeling pretext tasks, where great improvements in performanceon various downstream tasks have been achieved. However, we find these twoparadigms have different characteristics: (i) contrastive models aredata-hungry that suffer from a representation over-fitting issue; (ii)generative models have a data filling issue that shows inferior data scalingcapacity compared to contrastive models. This motivates us to learn 3Drepresentations by sharing the merits of both paradigms, which is non-trivialdue to the pattern difference between the two paradigms. In this paper, wepropose Contrast with Reconstruct (ReCon) that unifies these two paradigms.ReCon is trained to learn from both generative modeling teachers andsingle/cross-modal contrastive teachers through ensemble distillation, wherethe generative student guides the contrastive student. An encoder-decoder styleReCon-block is proposed that transfers knowledge through cross attention withstop-gradient, which avoids pretraining over-fitting and pattern differenceissues. ReCon achieves a new state-of-the-art in 3D representation learning,e.g., 91.26% accuracy on ScanObjectNN. Codes have been released athttps://github.com/qizekun/ReCon.