1. Tutorial Introduction

PE3R (Perception-Efficient 3D Reconstruction) is an innovative open source 3D reconstruction framework released by the xML Lab of the National University of Singapore (NUS) on March 10, 2025. It achieves efficient and intelligent scene modeling by integrating multimodal perception technology. The project is based on a number of cutting-edge computer vision research results. It only needs to input 2D images to quickly complete 3D scene reconstruction. On the RTX 3090 graphics card, the average reconstruction time for a single scene is only 2.3 minutes, which is more than 65% more efficient than traditional methods.

In terms of technical implementation, PE3R adopts a modular design architecture:

• Its core reconstruction engine is based on DUSt3R/MASt3R technology, achieving efficient conversion from 2D images to 3D point clouds.
• The visual perception module integrates the SAM/SAM2 series segmentation models to ensure accurate recognition and segmentation of scene objects, while supporting efficient deployment on mobile terminals through the MobileSAM optimized version.
• The semantic understanding layer uses the SigLIP visual language model, which gives the system zero-sample cross-scene understanding capabilities, and users can directly query specific objects through natural language commands.

The most groundbreaking innovation of this project lies in its two-level optimization algorithm:

• In the first stage, the MST (minimum spanning tree) algorithm is used for fast rough alignment.
• In the second stage, refined reconstruction is achieved by introducing the semantically constrained Bundle Adjustment.

This design not only ensures the reconstruction quality, but also controls the video memory usage within 6.2 GB, allowing the system to run smoothly on consumer-grade GPUs.PE3R: Perception-Efficient 3D Reconstruction".

The computing resources used in this tutorial are RTX 4090.

2. Project Examples

3. Operation steps

 1. After starting the container, click the API address to enter the web page

 2. Usage steps

Once you enter the website, you can start using

If "Bad Gateway" is displayed, it means the model is initializing. Since the model is large, please wait about 1-2 minutes and refresh the page.

Note:

• Image upload:
  • Please upload 2 to 8 pictures in as many directions and as clear as possible.
  • If the effect is not satisfactory, please increase the number of uploaded pictures or improve the quality of the pictures.
• Threshold: It is crucial to set the threshold properly - too high a threshold may lead to missed detections, while too low a threshold may lead to false detections, so it needs to be adjusted according to the actual situation.

4. Discussion

🖌️ If you see a high-quality project, please leave a message in the background to recommend it! In addition, we have also established a tutorial exchange group. Welcome friends to scan the QR code and remark [SD Tutorial] to join the group to discuss various technical issues and share application effects↓ 

Citation Information

The citation information for this project is as follows:

@article{hu2025pe3r,
  title={PE3R: Perception-Efficient 3D Reconstruction},
  author={Hu, Jie and Wang, Shizun and Wang, Xinchao},
  journal={arXiv preprint arXiv:2503.07507},
  year={2025}
}