Yutian Chen Shi Guo Tianshuo Yang Lihe Ding Xiuyuan Yu Jinwei Gu Tianfan Xue

Abstract

Reconstructing fast-dynamic scenes from multi-view videos is crucial forhigh-speed motion analysis and realistic 4D reconstruction. However, themajority of 4D capture systems are limited to frame rates below 30 FPS (framesper second), and a direct 4D reconstruction of high-speed motion from low FPSinput may lead to undesirable results. In this work, we propose a high-speed 4Dcapturing system only using low FPS cameras, through novel capturing andprocessing modules. On the capturing side, we propose an asynchronous capturescheme that increases the effective frame rate by staggering the start times ofcameras. By grouping cameras and leveraging a base frame rate of 25 FPS, ourmethod achieves an equivalent frame rate of 100-200 FPS without requiringspecialized high-speed cameras. On processing side, we also propose a novelgenerative model to fix artifacts caused by 4D sparse-view reconstruction, asasynchrony reduces the number of viewpoints at each timestamp. Specifically, wepropose to train a video-diffusion-based artifact-fix model for sparse 4Dreconstruction, which refines missing details, maintains temporal consistency,and improves overall reconstruction quality. Experimental results demonstratethat our method significantly enhances high-speed 4D reconstruction compared tosynchronous capture.