Image Correspondence Matching via Convolutional Neural Network (MSc)

Image Correspondence Matching via Convolutional Neural Network (MSc)

Correspondence matching is one of important processes in image-based 3D reconstruction pipelines, and usually it is employed the structure of motion (Sfm) or the Multi-ViewStereo (MVS) to match the spatial relations (likes geometry, semantic matching and etc.) amongs the multiple images. Typically, for example, in the initial step of SfM, feature detectors and descriptors such as SIFT find distinctive points in the input images, but SIFT has some flaws that makes it impossible to find enough correspondence, i.e. sharking object, non-Lambertian surfaces, weakly textured surfaces, and so on.

Recently, there are a lot of works: [2] uses random forest to classify the SIFT feature vectors then predicting the image key-points and matching those to the image pairs. However, due to the limitations of SIFT feature representation, this approach cannot replace SfM or MVS in general performance. [3] contains a work to measure learning similarity by using convolutional neural network, [4] proposes a deep learning framework for accurate visual correspondences and demonstrate its effectiveness for both geometric and semantic matching, and [5] proves that the correspondence can be learned by convolutional neural network.

In this thesis, you will find the correspondence matching by using deep learning technologies and

overcome the defects of SfM/MVS. Our goal is that replacing the parts of correspondence matching in image-based 3D reconstruction pipelines.


[1] Furukawa, Yasutaka, and Carlos Hernández. “Multi-view stereo: A tutorial.” *Foundations and Trends® in Computer Graphics and Vision* 9.1-2 (2015): 1-148.

[2] Hartmann, Wilfried, Michal Havlena, and Konrad Schindler. “Predicting matchability.” *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*. 2014.

[3] Zbontar, Jure, and Yann LeCun. “Stereo matching by training a convolutional neural network to compare image patches.” *Journal of Machine Learning Research* 17.1-32 (2016): 2.

[4] Choy, Christopher B., et al. “Universal correspondence network.” *Advances in Neural Information Processing Systems*. 2016.

[5] Long, Jonathan L., Ning Zhang, and Trevor Darrell. “Do convnets learn correspondence?.” *Advances in Neural Information Processing Systems*. 2014.