3D reconstruction of fruit trees and fruit semantic segmentation based on NeRF and improved RandLA-Net

Objective To solve the problem of accurate fruit segmentation in complex orchard environment.

Method A novel method for 3D reconstruction citrus fruit trees and fruit semantic segmentation of was proposed. First, the implicit 3D representation of the fruit tree was learned from multi-view images using the neural radiance field (NeRF) technology, generating high-quality point cloud models of the fruit tree. Then, the improved random local point cloud feature aggregation network (RandLA-Net) was adopted to conduct end-to-end semantic segmentation of the fruit tree point cloud, accurately extracting the fruit point cloud. In this study, targeted improvements were made to RandLA-Net. A bilateral enhancement module was added after the encoder layer, and a loss function more suitable for the fruit point cloud segmentation task was adopted. The improved segmentation network was verified using the citrus fruit tree dataset.

Result  The results showed that the proposed method could effectively reconstruct the 3D structure of fruit tree. The average intersection over union (mIoU) of the improved network increased by 2.64 percentage points, and the intersection over union (IoU) of the fruit increased by 7.33 percentage points, verifying the practicality of this method in the scenario of smart orchards.

Conclusion  This study provides a new technical support for achieving intelligent management and automated fruit harvesting in orchards.