Zeid Karim Abou ; Schult Jonas ; Hermans Alexander ; Leibe Bastian

Abstract

Recently, the self-supervised learning framework data2vec has shown inspiringperformance for various modalities using a masked student-teacher approach.However, it remains open whether such a framework generalizes to the uniquechallenges of 3D point clouds. To answer this question, we extend data2vec tothe point cloud domain and report encouraging results on several downstreamtasks. In an in-depth analysis, we discover that the leakage of positionalinformation reveals the overall object shape to the student even under heavymasking and thus hampers data2vec to learn strong representations for pointclouds. We address this 3D-specific shortcoming by proposing point2vec, whichunleashes the full potential of data2vec-like pre-training on point clouds. Ourexperiments show that point2vec outperforms other self-supervised methods onshape classification and few-shot learning on ModelNet40 and ScanObjectNN,while achieving competitive results on part segmentation on ShapeNetParts.These results suggest that the learned representations are strong andtransferable, highlighting point2vec as a promising direction forself-supervised learning of point cloud representations.