Non-rigid Structure from Motion and Multi-View Matching (BSc / MSc)

Thesis Introduction

Image-based, multi-view 3D reconstruction of dynamic scenes is a challenging task which our research group wants to tackle in the near future. The basic idea is that a 3D scene model (e.g. textured triangle mesh) is reconstructed for an image set from an uncontrolled capture setup. Uncontrolled setup hereby entails various challenges like deforming objects, changing illumination, varying and irregular scene surface sampling, unknown camera intrinsics and extrinsics etc. Our group developed the in-house 3D reconstruction pipeline MVE (abbreviation for Multi-View Environment) which handles most of the previously mentioned challenges. The pipeline separates the problem of image-based, multi-view 3D reconstruction into the following sub problems: Structure from Motion (SfM), Multi-View Stereo (MVS), Surface Reconstruction (SR) and Texture Reconstruction (TR), see Fig. 1.

Our SfM implementation takes as input a set of images and outputs image features (SIFT, SURF), matches the found features to compute corresponding feature tracks and finally estimates camera parameters and a sparse point cloud via these tracks to represent the capture process and the scene, see Fig. 2.

Having registered cameras and a sparse scene representation from SfM enables MVS to estimate a depth map containing per-pixel camera to scene distances for each image (with a registered camera). SR fuses those depth map estimates into a globally consistent 3D model. Eventually, TR reconstructs sharp, high-resolution and consistent surface textures for its input 3D scene model.

However, MVE cannot reconstruct dynamic scenes containing deforming objects, such as plants, cloth, persons, paper etc. The pipeline focuses on static scenes and does neither model rigidly moving nor deforming objects. The goal of this thesis is to extend our SfM and MVS steps to enable accurate and robust depth map reconstruction for dynamic scenes.

Related Work

SfM / capture (view/image/depth map) registration

– MVE – A Multi-View Reconstruction Environment

Fuhrmann, Simon and Langguth, Fabian and Goesele, Michael

GCH (Eurographics Workshops on Graphics and Cultural Heritage), 2014

– A Simple Prior-Free Method for Non-rigid Structure-from-Motion Factorization

Dai, Yuchao and Li, Hongdong and He, Mingyi

IJCV (International Journal of Computer Vision), 2014

– Kernel Non-Rigid Structure from Motion

Gotardo, Paulo F.U. and Aleix M. Martinez

ICCV (IEEE International Conference on Computer Vision), 2011

– Non-Rigid Structure from Motion with Complementary Rank-3 Spaces

Gotardo, Paulo F.U. and Aleix M. Martinez

CVPR (Computer Vision and Pattern Recognition), 2011

– Non-Rigid Structure from Locally-Rigid Motion

Taylor, Jonathan and Jepson, Allan D. and Kutulakos, Kiriakos N.

CVPR (Computer Vision and Pattern Recognition), 2010

– Good Vibrations: A Modal Analysis Approach for Sequential Non-Rigid Structure from Motion

Agudo, Antonio and Agapito, Lourdes and Calvo, Begona and Montiel, J. M. M.

CVPR (Computer Vision and Pattern Recognition), 2014

– Multibody Structure-from-Motion in Practice

Ozden Kemal Egemen and Schindler, Konrad and Gool, Luc Van

PAMI (IEEE Transactions on Pattern Analysis and Machine Intelligence), 2010

Multi-View Stereo

– Multi-View Stereo for Community Photo Collections

Goesele, Michael and Snavely, Noah and Curless, Brian and Hoppe, Hugues and Seitz, Steven M.

ICCV (IEEE International Conference on Computer Vision), 2007

– 3D Scene Flow Estimation with a Piecewise Rigid Scene Model

Vogel, Christoph and Schindler, Konrad and Roth, Stefan

IJCV (International Journal of Computer Vision), 2015

– Multi-View Stereo Reconstruction and Scene Flow Estimation with a Global Image-Based Matching Score

Pons, Jean-Philippe and Keriven, Renaud and Faugeras, Oliver

IJCV (International Journal of Computer Vision), 2006

– Dynamic Shape Capture using Multi-View Photometric Stereo

Vlasic, Daniel and Peers, Pieter and Baran, Ilya and Debevec, Paul and Popovic, Jovan


Surface Reconstruction / Depth Map Fusion

– DynamicFusion: Reconstruction and Tracking of Non-rigid Scenes in Real-Time

Newcombe, Richard A. and Fox, Dieter and Seitz, Steven M.

CVPR (Computer Vision and Pattern Recognition), 2015

– Multi-body Depth Map Fusion with Non-intersection Constraints

Jacquet Bastien, and Häne, Christian and Angst, Roland and Pollefeys, Marc

ECCV (European Conference on Computer Vision), 2014