Polina Karpikova Daniil Selikhanovych Kirill Struminsky Ruslan Musaev Maria Golitsyna Dmitry Baranchuk

Abstract

Recent advances in scene reconstruction have pushed toward highly realisticmodeling of autonomous driving (AD) environments using 3D Gaussian splatting.However, the resulting reconstructions remain closely tied to the originalobservations and struggle to support photorealistic synthesis of significantlyaltered or novel driving scenarios. This work introduces MADrive, amemory-augmented reconstruction framework designed to extend the capabilitiesof existing scene reconstruction methods by replacing observed vehicles withvisually similar 3D assets retrieved from a large-scale external memory bank.Specifically, we release MAD-Cars, a curated dataset of {sim}70K 360{\deg}car videos captured in the wild and present a retrieval module that finds themost similar car instances in the memory bank, reconstructs the corresponding3D assets from video, and integrates them into the target scene throughorientation alignment and relighting. The resulting replacements providecomplete multi-view representations of vehicles in the scene, enablingphotorealistic synthesis of substantially altered configurations, asdemonstrated in our experiments. Project page:https://yandex-research.github.io/madrive/