Living Scenes: Multi-object Relocalization
and Reconstruction in Changing 3D Environments
Liyuan Zhu*, Shengyu Huang+, Konrad Schindler+, Iro Armeni*
*Stanford University, +ETH Zurich
CVPR 2024 highlight
Abstract
Research into dynamic 3D scene understanding has primarily focused on short-term change tracking from dense observations, while little attention has been paid to long-term changes with sparse observations. We address this gap with MORE, a novel approach for multi-object relocalization and reconstruction in evolving environments. We view these environments as "living scenes" and consider the problem of transforming scans taken at different points in time into a 3D reconstruction of the object instances, whose accuracy and completeness increase over time.
At the core of our method lies an SE(3)-equivariant representation in a single encoder-decoder network, trained on synthetic data. This representation enables us to seamlessly tackle instance matching, registration, and reconstruction. We also introduce a joint optimization algorithm that facilitates the accumulation of point clouds originating from the same instance across multiple scans taken at different points in time. We validate our method on synthetic and real-world data and demonstrate state-of-the-art performance in both end-to-end performance and individual subtasks
Examples of living scenes: long-term changing 3D environments
Method Overview
Living Scenes. A living scene is a 3D environment with multiple moving objects that evolve over time. (a) Two temporal observations (scans) represent the scene at times (t1, t2) and capture the objects having moved around. To understand the change in the scene, given instance segmentation, we (b) match object point clouds from t1 and t2 that belong to the same instance, (c) register and reconstruct the matches through our joint optimization, (d) accumulate all point clouds per instance from the multiple temporal scans, improving the registration and reconstruction quality over time. We illustrate on two scans for simplicity.
Qualitative Results
Multi-object Relocalization
Multi-object Reconstruction
Citation
Please cite our paper if you find our method helpful
@inproceedings{zhu2023living,
author = {Liyuan Zhu and Shengyu Huang and Konrad Schindler, Iro Armeni},
title = {Living Scenes: Multi-object Relocalization and Reconstruction in Changing 3D Environments},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2024}
}