Lu Shiyang ; Chang Haonan ; Jing Eric Pu ; Boularias Abdeslam ; Bekris Kostas

Abstract

This work presents OVIR-3D, a straightforward yet effective method foropen-vocabulary 3D object instance retrieval without using any 3D data fortraining. Given a language query, the proposed method is able to return aranked set of 3D object instance segments based on the feature similarity ofthe instance and the text query. This is achieved by a multi-view fusion oftext-aligned 2D region proposals into 3D space, where the 2D region proposalnetwork could leverage 2D datasets, which are more accessible and typicallylarger than 3D datasets. The proposed fusion process is efficient as it can beperformed in real-time for most indoor 3D scenes and does not requireadditional training in 3D space. Experiments on public datasets and a realrobot show the effectiveness of the method and its potential for applicationsin robot navigation and manipulation.