Non-rigid Structure from Motion and Multi-View Matching (BSc / MSc)
Image-based, multi-view 3D reconstruction of dynamic scenes is a challenging task which our research group wants to tackle in the near future. The basic idea is that a 3D scene model (e.g. textured triangle mesh) is reconstructed for an image set from an uncontrolled capture setup. Uncontrolled setup hereby entails various challenges like deforming objects, changing illumination, varying and irregular scene surface sampling, unknown camera intrinsics and extrinsics etc. Our group developed the in-house 3D reconstruction pipeline MVE (abbreviation for Multi-View Environment) which handles most of the previously mentioned challenges. The pipeline separates the problem of image-based, multi-view 3D reconstruction into the following sub problems: Structure from Motion (SfM), Multi-View Stereo (MVS), Surface Reconstruction (SR) and Texture Reconstruction (TR), see Fig. 1.
Our SfM implementation takes as input a set of images and outputs image features (SIFT, SURF), matches the found features to compute corresponding feature tracks and finally estimates camera parameters and a sparse point cloud via these tracks to represent the capture process and the scene, see Fig. 2.
Having registered cameras and a sparse scene representation from SfM enables MVS to estimate a depth map containing per-pixel camera to scene distances for each image (with a registered camera). SR fuses those depth map estimates into a globally consistent 3D model. Eventually, TR reconstructs sharp, high-resolution and consistent surface textures for its input 3D scene model.
However, MVE cannot reconstruct dynamic scenes containing deforming objects, such as plants, cloth, persons, paper etc. The pipeline focuses on static scenes and does neither model rigidly moving nor deforming objects. The goal of this thesis is to extend our SfM and MVS steps to enable accurate and robust depth map reconstruction for dynamic scenes.
SfM / capture (view/image/depth map) registration
– MVE – A Multi-View Reconstruction Environment
Fuhrmann, Simon and Langguth, Fabian and Goesele, Michael
GCH (Eurographics Workshops on Graphics and Cultural Heritage), 2014
– A Simple Prior-Free Method for Non-rigid Structure-from-Motion Factorization
Dai, Yuchao and Li, Hongdong and He, Mingyi
IJCV (International Journal of Computer Vision), 2014
– Kernel Non-Rigid Structure from Motion
Gotardo, Paulo F.U. and Aleix M. Martinez
ICCV (IEEE International Conference on Computer Vision), 2011
– Non-Rigid Structure from Motion with Complementary Rank-3 Spaces
Gotardo, Paulo F.U. and Aleix M. Martinez
CVPR (Computer Vision and Pattern Recognition), 2011
– Non-Rigid Structure from Locally-Rigid Motion
Taylor, Jonathan and Jepson, Allan D. and Kutulakos, Kiriakos N.
CVPR (Computer Vision and Pattern Recognition), 2010
– Good Vibrations: A Modal Analysis Approach for Sequential Non-Rigid Structure from Motion
Agudo, Antonio and Agapito, Lourdes and Calvo, Begona and Montiel, J. M. M.
CVPR (Computer Vision and Pattern Recognition), 2014
– Multibody Structure-from-Motion in Practice
Ozden Kemal Egemen and Schindler, Konrad and Gool, Luc Van
PAMI (IEEE Transactions on Pattern Analysis and Machine Intelligence), 2010
– Multi-View Stereo for Community Photo Collections
Goesele, Michael and Snavely, Noah and Curless, Brian and Hoppe, Hugues and Seitz, Steven M.
ICCV (IEEE International Conference on Computer Vision), 2007
– 3D Scene Flow Estimation with a Piecewise Rigid Scene Model
Vogel, Christoph and Schindler, Konrad and Roth, Stefan
IJCV (International Journal of Computer Vision), 2015
– Multi-View Stereo Reconstruction and Scene Flow Estimation with a Global Image-Based Matching Score
Pons, Jean-Philippe and Keriven, Renaud and Faugeras, Oliver
IJCV (International Journal of Computer Vision), 2006
– Dynamic Shape Capture using Multi-View Photometric Stereo
Vlasic, Daniel and Peers, Pieter and Baran, Ilya and Debevec, Paul and Popovic, Jovan
ACM SIGGRAPH Asia, 2009
Surface Reconstruction / Depth Map Fusion
– DynamicFusion: Reconstruction and Tracking of Non-rigid Scenes in Real-Time
Newcombe, Richard A. and Fox, Dieter and Seitz, Steven M.
CVPR (Computer Vision and Pattern Recognition), 2015
– Multi-body Depth Map Fusion with Non-intersection Constraints
Jacquet Bastien, and Häne, Christian and Angst, Roland and Pollefeys, Marc
ECCV (European Conference on Computer Vision), 2014