Project Lab Capturing Reality (Projektpraktikum Capturing Reality)

Project Lab Capturing Reality (Projektpraktikum Capturing Reality)

In this project lab groups of students work on selected large topics in capturing reality, i.e., on the boundary between computer vision and computer graphics. Project results will be presented in a talk at the end of the course. The specific topics addressed in the project lab change every semester.

Modalities (Credit Points, Prerequisites, …)

For all information regarding modalities of this lecture please have a look into TUCaN.

Topics Winter Term 2017/18

We offer one highly interesting and hands-on group project this semester!

Structure from Motion with Markov Random Fields

Introduction
Our research group heavily uses the inhouse reconstruction pipeline called Multi-View Environment (MVE).The MVE pipeline usually involves the following steps:

  • Structure from Motion (SfM) to estimate camera parameters and a sparse point cloud of the scene
  • Multi-View Stereo (MVS) to obtain depth maps for each camera and a dense point cloud for scene representation
  • Surface Reconstruction (e.g., FSSR, TSR) to compute surface triangle meshes from dense point clouds
  • Post processing of surface meshes, e.g., texture reconstruction

Motivation
The first step (SfM) of our reconstruction pipeline is already challenging. SfM thus usually involves the following complex sub steps. First, features are detected in each image (low dimensional representations of distinctive image areas). Second, these features are matched between image pairs to estimate camera parameters for pairs of images. Extrinsic (relative rotational and translational offset) and intrinsic parameters (focal length, etc.) of two images can be estimated by means of enough feature point correspondences between them. Incremental SfM approaches begin with a random pair of images and then add one image after the other to the set of registered cameras via the point correspondences per image pair. The incremental registering of cameras is problematic as it is computationally intensive and not robust. For this reason novel approaches consider global formulations of SfM. Meaning they estimate camera parameters for all images equally at one time by fulfilling a set of constraints defined on the complete image set.

Task
The goal of this project lab is to implement and integrate a novel and global SfM technique in a small group of students. The core of the novel technique is based on a Markov Random Field (MRF) formulation defining pairwise constraints between images or feature points and images as well as unary constraints for single images when additional information is available, such as geotags or vanishing points. The students are expected to implement and integrate the approach described in the following publication:
SfM with MRFs: Discrete-Continuous Optimization for Large-Scale Structure from Motion
David Crandall and Andrew Owens and Noah Snavely and Daniel Huttenlocher
In: IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2013.
PDF

To properly integrate the novel SfM technique into our reconstruction pipeline MVE, the lab attendees are required to get familiar with MVE. Eventually, they need to make use of the features already provided by MVE and adapt the techniques of the paper in order to make them work well in conjunction with MVE. This might involve additional steps to be implemented which are not mentioned in the publication and will become clear during the course. This means that the lab not only includes understanding of state-of-the-art reconstruction approaches and programming but also a substantial amount of work concerning planning and discussions with the GCC employees.

At last, the students are supposed to run their implemention on multiple datasets (including given large scale image sets) to have results for evaluation of the strengths and weaknesses of the novel SfM technique.

For more information regarding the project have a look at the official project web page:
http://vision.soic.indiana.edu/projects/disco/

Highly recommended preknowledge:

  • lecture “Capturing Reality”
  • lecture “Computer Vision I”
  • lecture “Computer Graphics I”
  • C++

Media:
See the following official video from the authors' project web page mentioned above:

Topics Winter Term 2016/17

We offer a highly interesting and hands-on project this semester!

Occluding Contours For Multi-View Stereo

Introduction
The GCC research members heavily use their inhouse reconstruction pipeline called Multi-View Environment (MVE). Our MVE pipeline usually involves the following steps:

  • Structure from Motion (SfM) to register cameras and get a sparse point cloud
  • Multi-View Stereo (MVS) to obtain depth maps for each camera and a dense point cloud for scene representation
  • Floating Scale Surface Reconstruction (FSSR) to compute a surface triangle mesh from the dense point cloud
  • Post processing of the surface mesh, e.g. simplification

Motivation
Reconstruction of edges, corners or other sharp features are sometimes not accurate using our MVS technique and FSSR. For example, reconstructed meshes of roof or tree parts might falsely include parts of the sky or surrounding scene.

Task
The goal of this project lab is to implement and integrate a novel approach based on so called occluding contours (visibility constraints) in order to provide high quality results for the mentioned difficult cases as well. The students are expected to implement the techniques of the following publication:
Shan, Qi et al. “Occluding Contours for Multi-View Stereo” CVPR. 2014.
Furthermore, the lab attendees are required to get familiar with our reconstruction pipeline MVE. Eventually, they need to adapt the techniques of the paper in order to make them work well in conjunction with MVE. This might involve additional steps to be implemented which are not mentioned in the publication and will become clear during the course. Most likely, it will be necessary to make the approach work w.r.t. multi-scale input data. This means that the lab not only includes understanding of state-of-the-art reconstruction approaches and programming but also a substantial amount of work concerning planning and discussions with the GCC employees.

For more information regarding the project you can also have a look at the University of Washington's project web page: http://grail.cs.washington.edu/projects/sq_rome_g2/


Highly recommended preknowledge:

  • lecture “Capturing Reality”
  • lecture “Computer Vision I”
  • lecture “Computer Graphics I”
  • C++

Media:

Vorschaubild

Topics Winter Term 2015/2016

We offer two highly interesting and hands-on projects this semester!

Fast Image Matching

Introduction
The GCC research members heavily use their inhouse reconstruction pipeline called Multi-View Environment (MVE). The usual first step for multi-view reconstruction of visual 3D models from photos is called Structure from Motion (SfM) and involves the detection of distinguishable features and their matching. Since this matching process must be done for image pairs its runtime requirements are high for datasets with a large number of photos when implemented naively.

The goal of this project lab is to realize a novel and very efficient approach that reduces the complexity of image matching to speed up the reconstruction process of MVE. The approach is based on the publication:
Heinly, Jared, et al. “Reconstructing the World* in Six Days*(As Captured by the Yahoo 100 Million Image Dataset).” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015.


Highly recommended preknowledge:

  • lecture “Capturing Reality”
  • lecture “Computer Vision I”
  • C++

Official website:
http://www.cs.unc.edu/~jheinly/reconstructing_the_world.html

Media:

Vorschaubild

CMPMVS

Introduction
There are several hard cases which are difficult to be reconstructed when using a passive multi-view capturing approach, such as photo datasets of scenes with foliage or translucent objects. The goal of this project is to implement a robust reconstruction algorithm that can better handle such datasets despite the difficult detection of distinguishable and correctly matchable image features. The goal is to implement the algorithm of the publication:
Jancosek, Michal, and Tomás Pajdla. “Multi-view reconstruction preserving weakly-supported surfaces.” Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on. IEEE, 2011..


Highly recommended preknowledge:

  • lecture “Capturing Reality”
  • lecture “Computer Graphics I (GDV I)”
  • lecture “Computer Graphics II (GDV II)”
  • C++

Official website:
http://ptak.felk.cvut.cz/sfmservice/websfm.pl?menu=cmpmvs

Media:

Vorschaubild