How LiDAR Scans and RGB Images Can Help Komatsu Create a Tree and Obstacle Recognition Model for Improved Forest Harvesting
Komatsu had an ambitious vision for improving forestry productivity - they wanted to find a more sustainable approach to planning and executing harvesting operations. Their goal was to create a reconstruction of the forest that would help them identify trees ready for harvest and optimize the route for heavy machinery by providing information on the location of protected objects and terrain specifications.
We’ve Set Out to Create a Computer Vision System for Identifying Objects and Telling their Location
Manual methods of forest mapping are slow, labor-intensive, and error-prone. To address these challenges, we proposed creating a machine learning model, which is ideal for quickly and accurately processing large, complex data sets. In a forest environment, where trees, terrain, and obstacles are often hidden or obscured, ML algorithms can automatically detect patterns in the data, identify different tree species, and locate obstacles with high accuracy.
đź‘‹ Hey You, What Are You Working On?
Share a few details about your challenge or idea, no matter how early or complex. Our expert team will reach out to offer our expertise in terms of feasibility, budget, and scope. Let’s shape a solution for you together!
A Drone 3D Scanning of the Terrain Can Help Obtain the Highest Quality Dataset
Could it be that a forest didn’t have enough trees? In this case, it was! Our first challenge was the lack of reliable dataset to train the machine learning model. Komatsu provided us with photos of the forest. However, the images were too blurry for us to use and did not include enough obstacles for us to train machine learning algorithms.
Our team built and used a drone to collect LiDAR scans and RGB images of the forest. LiDAR data can be used to accurately measure the size, shape, and location of objects in 3D space, necessary for training the algorithm to identify objects that might be challenging to detect using RGB images alone, such as trees and objects that were partially obscured by foliage. The RGB images, on the other hand, were deemed to prove useful for more detailed classification based on texture and color, useful when distinguishing between a tree and obstacle.
Project delivered in partnership with:
Equipment used for the test flight:
- Drone that was built by our team and equipped with a LiDAR sensor and RGB camera.
Data we wanted to obtain:
- Geolocation data for the precise position of the drone during each LiDAR scan and RGB image capture.
- LiDAR data to obtain a detailed 3D point cloud of the forestry environment.
- High-resolution images from the RGB camera to capture visual details.
The Idea Was to Clean the Data and Conduct Fusion of the Two Datasets for Training ML Algorithm
After collecting raw images, we applied advanced machine learning algorithms to calibrate and correct any distortions, ensuring the accuracy and reliability of the data. However, to create a comprehensive dataset, we still needed to integrate the RGB images with the LiDAR data.
This integration process involved two key steps – time synchronization to ensure the data from both sensors were perfectly aligned, and transformation and rotation to match LiDAR points with objects in the RGB images. This was necessary due to LiDAR’s nearly 360-degree field of view, in contrast to the narrower field of view of the RGB camera.
Finally, as part of an experiment, we filtered the data to retain only the points corresponding to trees, and we discarded irrelevant information while also isolating terrain points for further analyses.
Tree Measurement and DBH Calculation for Better Harvesting
When mapping trees ready for harvesting, it’s essential to calculate the Diameter at Breast Height (DBH) for each tree. DBH is a critical metric for assessing tree health, assessing its quality for sustainable harvesting and more informed decision-making. Using the integrated LiDAR and RGB data, we partialy developed experimental algorithms that are supposed to estimate the DBH based on the tree’s size and structure in the future..
The Solution Will Use Semantic Segmentation to Split Objects Apart
To map out the area, we want to use geolocation data recorded simultaneously with the LiDAR scans and RGB imagery. This would enable us to calculate the precise positions of various objects.
We will apply semantic segmentation, a sophisticated machine learning technique, to identify specific objects of interest, such as trees and obstacles, and determine their locations.
We plan to utilize the Segment Anything Model in order to refine the segmentation results and produce smooth, precise masks. SAM is an adaptive segmentation system that will ensure the masks were sharp and highly accurate.
The result? Our model will be able to recognize trees and other objects together with their geolocalization and required parameters.
The Final Solution Will Enable Tree Species Identification and Realistic Surface Mapping with Drones
By combining LiDAR data, RGB imagery, machine learning, and semantic segmentation, we plan to create a robust computer vision tree and obstacle recognition system that supports Komatsu’s efforts to streamline forest harvesting while minimizing environmental impact.
By accurately mapping tree locations and obstacles such as rocks, stumps, and protected areas, the model will enable efficient and safe path planning for machines, reducing the risk of environmental damage and ensuring minimal disruption to the ecosystem.
The final solution will be ready to optimize Komatsu’s operations in a variety of areas:
- Analysis of trees, their type and exact location.
- Route detection and optimization for heavy equipment
- Optimize harvesting to maximize volume and quality of timber and reduce the resources required for harvesting.
- Environmental and cultural protection, as our algorithm detects protected places such as anthills or graves.
- The map provided by our solutions is more accurate and accessible than those provided by government agencies.
The technology can be used in real-time, so live events, manufacturing halls, busy streets, and large areas can also be monitored.
About the Grant
The case study was developed based on the Agrarsense project under Grant Agreement No. 101095835, supported by the Chips Joint Undertaking and its members, including the top-up funding by Sweden, Czechia, Finland, Ireland, Italy, Latvia, Netherlands, Norway, Poland, and Spain, including the National Centre for Research and Development of Poland. Project is co-funded by the Chips Joint Undertaking and the National Center for Research and Development.
Got a Project in Mind? Let’s Discuss It.
Contact us!
Send us an email: [email protected]
Looking for Machine Vision Solutions for Your Business?
