How cutting-edge technology transforms images into 3D reality

Reconstructing 3D objects and buildings from images: DAC.digital’s cutting-edge research-science project.

Outline of the problem

The key issue is to reconstruct 3D objects and buildings from unstructured image collections freely available on the internet. The challenge is to identify which parts of two images capture the same physical points of a scene, establish correspondences between pixel coordinates of image locations, and recover the 3D location of points by triangulation.

Proposed solution

The proposed solution is to develop a machine learning algorithm based on computer vision techniques to register two images from different viewpoints. By creating a method to identify key points in the images and establish correspondences between them, we can calculate the fundamental matrix, which provides essential information about where and from which viewpoints the photos were taken. This process will lead to the generation of 3D models of the landmarks.

Which technologies have we applied:

Python,
Pytorch,
Kornia,
OpenCV

Connecting Perspectives: Algorithmic Solutions for Cross-View Landmark Recognition in Tourist Imagery.

There were collections of tourist images of 16 landmarks taken from various angles and distances, such as nearby, below, and sometimes with obstructions like people. The challenge was to develop algorithms capable of identifying key points in these images (located on buildings) and then establish the correspondences between them across different viewpoints, even without knowing the exact camera parameters used to capture the images. The difficulty lies in dealing with diverse viewpoints, lighting conditions, occlusions, and user-applied filters in the images, without having access to capture location and device parameters like camera models and lenses.

The people and tech behind our project.

The team consisted of five developers and researchers with varying levels of experience in computer vision, machine learning, and image processing.

Each member was responsible for working independently in their niche area and performing experiments while also collaborating and discussing progress with others to finally integrate best approaches in one system.

The project leveraged Python for scripting and building experiment architecture. PyTorch was used to build and train neural networks for keypoint detection and matching, while Kornia provided state-of-the-art models for Computer Vision. OpenCV handled image preprocessing and image manipulation tasks. This cohesive tech stack enabled efficient experimentation and remarkable progress in 3D object reconstruction from diverse image collections.

A Holistic Journey through Landmark Recognition: From Exploration to Validation.

Unveiling Insights Through Literature Exploration

First, we delved into an in-depth literature review to gain a comprehensive understanding of existing solutions and techniques in the realm of stereophotogrammetry and 3D reconstruction from images. This initial phase allowed us to grasp the state-of-the-art approaches and identify potential areas for improvement.

Navigating Algorithms and Models for Key Point Identification

Next, we proceeded with experimentation, exploring various computer vision algorithms and machine learning models. Our primary aim was to identify key points within the images and establish meaningful correspondences across different viewpoints. This experimental stage enabled us to assess the performance and limitations of different approaches, guiding us towards the most promising paths.

From Theory to Reality: Prototyping and Refinement

With valuable insights from the experimentation phase, we moved on to developing prototypes. These prototypes served as crucial testing grounds for implementing diverse algorithms and fine-tuning parameter combinations on our dataset. Through this iterative process, we gained valuable feedback and refined our methods.

Forging Cohesion: Seamlessly Merging Algorithms and Techniques

As the project’s complexity demanded an integration of various algorithms and techniques, we dedicated substantial effort to ensuring a seamless fusion of components. This integration phase required meticulous coordination and harmonization of different modules to ensure they functioned cohesively.

Testing the Waters: Evaluating Performance and Potential

Finally, we put our solution to the test. Through extensive testing on unseen data, we rigorously evaluated its performance, assessing its accuracy and generalizability. This thorough examination allowed us to validate the effectiveness of our approach and ascertain its potential for real-world applications.

Results

The team’s developed machine learning algorithm successfully registered images from different viewpoints and calculated the fundamental matrix. This allowed them to create accurate 3D models of the landmarks from the collections of tourist images.

Key numbers

The project achieved success in solving the complex computer vision problem in a relatively short time frame of slightly over one month.

The proposed solution showed promising results and had potential applications in virtual and augmented reality, cultural heritage preservation, and others 3D reconstructions where the data is not complete.

Michał Affek Embedded Machine Learning Researcher

Pushing Limits in Computer Vision: Join Our Journey of Innovation.

Computer vision topics can be both challenging and innovative, as demonstrated by DAC.digital’s remarkable research-science project in reconstructing 3D objects and buildings from images. If you are interested in embarking on a project that involves computer vision and pushing the boundaries of this cutting-edge technology, we invite you to contact us to collaborate and work together!

Let’s join forces to unlock new possibilities in the world of computer vision.