High Fidelity 3D Reconstructions
with Limited Physical Views

Mosam Dabhi1, Chaoyang Wang1, Kunal Saluja2, László A. Jeni1, Ian R. Fasel2, Simon Lucey1,3
1Carnegie Mellon University, 2Apple, 3The University of Adelaide

International Conference on 3D Vision - 3DV, 2021
Paper arXiv Video (long) Video (short)
Code Poster Feature Post Tweet

Overview of Neural Prior based MV-NRSfM

Enforcing Neural "shape" Priors and multi-view equivariance within modern deep 2D-3D lifting enables generation of high-fidelity 3D reconstructions using just 2-3 uncalibrated cameras, compared to >100 calibrated cameras.

Abstract

Multi-view triangulation is the gold standard for 3D reconstruction from 2D correspondences given known calibration and sufficient views. However in practice, expensive multi-view setups – involving tens sometimes hundreds of cameras – are required in order to obtain the high fidelity 3D reconstructions necessary for many modern applications. In this paper we present a novel approach that leverages recent advances in 2D-3D lifting using neural shape priors while also enforcing multi-view equivariance - and term this approach Multi-view Non-Rigid Structure-from-Motion (MV-NRSfM). We show how our method can achieve comparable fidelity to expensive calibrated multi-view rigs using a limited (2-3) number of uncalibrated camera views.

Deep-dive video

More Results

We show results and baseline comparisons on three datasets. For all the datasets, we compare against triangulation that uses a substantially large (~ 30-50 for monkey dataset) number of calibrated camera views. We also compare against the single view NRSfM versus multi-view NRSfM (ours). Please see the paper for more details.

BibTeX

@inproceedings{dabhi20213dv,
  author      = {Dabhi, Mosam and Wang, Chaoyang and Saluja, Kunal and Jeni, Laszlo and Fasel, Ian and Lucey, Simon},
  title       = {High Fidelity 3D Reconstructions with Limited Physical Views},
  booktitle   = {International Conference on 3D Vision (3DV)},
  year        = {2021},
}