3D reconstruction and semantic segmentation of fruit trees based on NeRF and improved RandLA-Net

YANG Dacheng; GUO Jun; YANG Jing; ZHANG Yali; LAN Yubin

doi:10.7671/j.issn.1001-411X.202410015

Journal of South China Agricultural University > 2025 > Uncorrected proof > DOI: 10.7671/j.issn.1001-411X.202410015

YANG Dacheng, GUO Jun, YANG Jing, et al. 3D reconstruction and semantic segmentation of fruit trees based on NeRF and improved RandLA-Net[J]. Journal of South China Agricultural University, 2025, 46(4): 1-10. DOI: 10.7671/j.issn.1001-411X.202410015

Citation:

PDF (2322 KB)

3D reconstruction and semantic segmentation of fruit trees based on NeRF and improved RandLA-Net

YANG Dacheng^{1, 2,},
GUO Jun^{1, 2},
YANG Jing^{1, 2},
ZHANG Yali^{1, 2, ,},
LAN Yubin^{2, 3, ,}

1.
College of Engineering, South China Agricultural University, Guangzhou 510642, China
2.
National Center for International Collaboration Research on Precision Agricultural Aviation Pesticide Spraying Technology, Guangzhou 510642, China
3.
College of Electronic Engineering & College of Artificial Intelligence, Guangzhou 510642, China

More Information

Graphical Abstract

Abstract

Abstract

Objective
To solve the problem of accurate fruit segmentation in complex orchard environment.
Method
A novel method for 3D reconstruction and semantic segmentation of citrus fruit trees was proposed. First, the implicit 3D representation of the fruit tree was learned from multi-view images using the neural radiance field (NeRF) technology, generating high-quality point cloud models of the fruit tree. Then, the improved random local point cloud feature aggregation network (RandLA-Net) was adopted to conduct end-to-end semantic segmentation of the fruit tree point cloud, accurately extracting the fruit point cloud. In this study, targeted improvements were made to RandLA-Net. A bilateral enhancement module was added after the encoder layer, and a loss function more suitable for the fruit point cloud segmentation task was adopted. The improved segmentation network was verified through experiments using the citrus fruit tree dataset.
Result
The results showed that the proposed method could effectively reconstruct the 3D structure of the fruit tree. The average intersection over union (mIoU) of the improved network increased by 2.64 percentage points, and the intersection over union (IoU) of the fruit increased by 7.33 percentage points, verifying the practicality of this method in the scenario of smart orchards.
Conclusion
This study provides new technical support for achieving intelligent management and automated fruit harvesting in orchards.
- Fruit tree reconstruction,
- Fruit segmentation,
- Neural radiance field(NeRF),
- Point cloud semantic segmentation,
- Smart agriculture

FullText(HTML)

References (35)

References

[1]	SANGIORGIO P, VERARDI A, SPAGNOLETTA A, et al. Citrus as a multifunctional crop to promote new bio-products and valorize the supply chain[J]. Environmental Engineering and Management Journal, 2020, 19(10): 1869-1889. doi: 10.30638/eemj.2020.179
[2]	肖阳, 项明宇, 李熹. 基于改进YOLOv8n的轻量化柑橘成熟度检测[J]. 计算机系统应用, 2024, 33(11): 202-208.
[3]	伍倩. 基于Kinect v2相机的柑橘点云分割与配准研究[D]. 桂林: 广西师范大学, 2022.
[4]	邹密. 基于三维重建的柑橘冠层特征检测系统设计与实现[D]. 重庆: 西南大学, 2023.
[5]	韩旭洁. 基于深度学习的果实点云分类与分割[D]. 杨凌: 西北农林科技大学, 2022.
[6]	AFONSO M, FONTEIJN H, FIORENTIN F S, et al. Tomato fruit detection and counting in greenhouses using deep learning[J]. Frontiers in Plant Science, 2020(11): 571299. doi: 10.3389/fpls.2020.571299.
[7]	PENG H, XUE C, SHAO Y, et al. Semantic segmentation of Litchi branches using DeepLabV³⁺ model[J]. IEEE Access, 2020(8): 164546-164555.
[8]	KANG H, CHEN C. Fruit detection and segmentation for apple harvesting using visual sensor in orchards[J]. Sensors, 2019, 19(20): 4599. doi: 10.3390/s19204599
[9]	BARGOTI S, UNDERWOOD J P. Image segmentation for fruit detection and yield estimation in apple orchards[J]. Journal of Field Robotics, 2017, 34(6): 1039-1060. doi: 10.1002/rob.21699
[10]	FU K, WEI P, VILLACRES J, et al. Fusion-driven tree reconstruction and fruit localization: Advancing precision in agriculture[EB/OL]. arXiv: 2310.15138(2023-10-23)[2024-10-01]. https://arxiv.org/abs/2310.15138v2.
[11]	GE Y, XIONG Y, FROM P J. Symmetry-based 3D shape completion for fruit localisation for harvesting robots[J]. Biosystems Engineering, 2020, 197: 188-202. doi: 10.1016/j.biosystemseng.2020.07.003
[12]	ZHAO L, HUANG S, DISSANAYAKE G. Linear SFM: A hierarchical approach to solving structure-from-motion problems by decoupling the linear and nonlinear components[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 141: 275-289. doi: 10.1016/j.isprsjprs.2018.04.007
[13]	SEITZ S M, CURLESS B, DIEBEL J, et al. A comparison and evaluation of multi-view stereo reconstruction algorithms[C]//2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06). New York, NY, USA: IEEE, 2006: 519-528.
[14]	DONG J, BURNHAM J G, BOOTS B, et al. 4D crop monitoring: Spatio-temporal reconstruction for agriculture[C]//2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore: IEEE, 2017: 3878-3885.
[15]	LI Z, ZHANG Z, LUO S, et al. An improved matting-SfM algorithm for 3D reconstruction of self-rotating objects[J]. Mathematics, 2022, 10(16): 2892. doi: 10.3390/math10162892.
[16]	GENÉ-MOLA J, GREGORIO E, CHEEIN F A, et al. Fruit detection, yield prediction and canopy geometric characterization using LiDAR with forced air flow[J]. Computers and Electronics in Agriculture, 2019, 168: 105121. doi: 10.1016/j.compag.2019.105121.
[17]	CHAKRABORTY M, KHOT L R, SANKARAN S, et al. Evaluation of mobile 3D light detection and ranging based canopy mapping system for tree fruit crops[J]. Computers and Electronics in Agriculture, 2019, 158: 284-293. doi: 10.1016/j.compag.2019.02.012
[18]	CAO W, WU J, SHI Y, et al. Restoration of individual tree missing point cloud based on local features of point cloud[J]. Remote Sensing, 2022(14): 1346. doi: 10.3390/rs14061346.
[19]	AI M, YAO Y, HU Q, et al. An automatic tree skeleton extraction approach based on multi-view slicing using terrestrial LiDAR scans data[J]. Remote Sensing, 2020, 12(22): 3824. doi: 10.3390/RS12223824.
[20]	HUANG S, GOJCIC Z, HUANG J, et al. Dynamic 3D scene analysis by point cloud accumulation[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 674-690.
[21]	ZAPATA N T, TSOULIAS N, SAHA K K, et al. Fourier analysis of LiDAR scanned 3D point cloud data for surface reconstruction and fruit size estimation[C]//2022 IEEE Workshop on Metrology for Agriculture and Forestry (MetroAgriFor). Perugia, Italy: IEEE, 2022: 197-202.
[22]	MILDENHALL B, SRINIVASAN P P, TANCIK M, et al. NeRF: Representing scenes as neural radiance fields for view synthesis[C]//Computer Vision, Cham: Springer International Publishing, 2020: 99-106.
[23]	ADAMKIEWICZ M, CHEN T, CACCAVALE A, et al. Vision-only robot navigation in a neural radiance world[J]. IEEE Robotics and Automation Letters, 2022, 7(2): 4506-4613.
[24]	SHEN S, WANG Z, LIU P, et al. Non-line-of-sight imaging via neural transient fields[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(7): 2257-2268. doi: 10.1109/TPAMI.2021.3076062
[25]	HONG K, WANG H, YUAN B. Inspection-Nerf: Rendering multi-type local images for dam surface inspection task using climbing robot and neural radiance field[J]. Buildings, 2023, 13(1): 213. doi: 10.3390/buildings13010213.
[26]	GAO X, YANG J, KIM J, et al. MPS-NeRF: Generalizable 3D human rendering from multiview images[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(12): 9154-9167.
[27]	DENG N, HE Z, YE J, et al. FoV-NeRF: Foveated neural radiance fields for virtual reality[J]. IEEE Transactions on Visualization and Computer Graphics, 2022, 28(11): 3854-3864. doi: 10.1109/TVCG.2022.3203102
[28]	QI C R, SU H, MO K, et al. PointNet: Deep learning on point sets for 3D classification and segmentation[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA: IEEE, 2017: 652-660.
[29]	QI C R, YI L, SU H, et al. PointNet++: Deep hierarchical feature learning on point sets in a metric space[C]//31st Conference on Neural Information Processing Systems (NeurIPS 2017), Long Beach, CA: NeurIPS, 2017: 5099-5108.
[30]	SHI S, WANG X, LI H. PointRCNN: 3D object proposal generation and detection from point cloud[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA: IEEE, 2019: 770-779.
[31]	SHI S, GUO C, JIANG L, et al. PV-RCNN: point-voxel feature set abstraction for 3D object detection[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA: IEEE, 2020: 10529-10538.
[32]	DING Z, HAN X, NIETHAMMER M. VoteNet: A deep learning label fusion method for multi-atlas segmentation[J]. Medical Image Computing and Computer-Assisted Intervention, 2019, 11766: 202-210.
[33]	HU Q, YANG B, XIE L, et al. RandLA-Net: Efficient semantic segmentation of large-scale point clouds[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA: IEEE, 2020: 11105-11114.
[34]	CUI Y, CHANG Q, LIU Q, et al. 3D reconstruction with spherical cameras[J]. IEEE Access, 2021, 9: 143531-143544. doi: 10.1109/ACCESS.2021.3119367.
[35]	钱德宇. 可移动文物数字化原真采集系统研究与实现[D]. 北京: 北京邮电大学, 2023.

[1]	MAI Guiwan, XIAO Wenchao, LI Yunfeng, LI Huaping, RAO Xueqin. Molecular identification and pathogenicity analysis of bacterial sheath rot pathogen in different types of banana[J]. Journal of South China Agricultural University, 2024, 45(4): 535-541. DOI: 10.7671/j.issn.1001-411X.202312018
[2]	JIA Jia, WANG Huan, DUAN Mingming, CHEN Bo, CHENG Yanbo, NIAN Hai. Accurate identification and correlation analysis of iron and zinc contents in soybean core accessions[J]. Journal of South China Agricultural University, 2023, 44(5): 760-768. DOI: 10.7671/j.issn.1001-411X.202211004
[3]	LI Youjia, YANG Fan, LÜ Baoqian, HE Rongxiao, CAI Bo. Analysis of haplotype diversity of an invasive leaf-eating pest Opisina arenosella[J]. Journal of South China Agricultural University, 2020, 41(4): 76-81. DOI: 10.7671/j.issn.1001-411X.201910023
[4]	HONG Malin, FU Cheng, HUANG San, ZHOU Pei, SU Shuo, LI Shoujun. Isolation, identification and sequence analysis of complete genome of feline panleukopenia virus in Guangzhou area[J]. Journal of South China Agricultural University, 2017, 38(1): 9-14. DOI: 10.7671/j.issn.1001-411X.2017.01.003
[5]	XIA Hongmei, LI Zhiwei, ZHEN Wenbin, ZHANG Bingchao. A simulation analysis on vacuum flow field for an air chamber of pneumatic plate-type vegetable seed metering device[J]. Journal of South China Agricultural University, 2014, 35(6): 99-103. DOI: 10.7671/j.issn.1001-411X.2014.06.019
[6]	XIE Li, LIU Fang, YI Maosheng, ZENG Ruizhen, XIA Qing, LI Yanghui, ZHANG Zhisheng. A correlation between ploidy levels and characters of leaf and flower in Phalaenopsis[J]. Journal of South China Agricultural University, 2014, 35(5): 82-87. DOI: 10.7671/j.issn.1001-411X.2014.05.014
[7]	CHEN Ting~1,YE Qing-sheng~2,LIU Wei~3. The orthogonal test of induction and proliferation of Dendrobium nobile protocrom like bodies(PLBs)[J]. Journal of South China Agricultural University, 2005, 26(3): 60-63. DOI: 10.7671/j.issn.1001-411X.2005.03.017
[8]	Adventitious Shoot Induction and Plant Regeneration from the Epicotyl of Arachis hypogaea[J]. Journal of South China Agricultural University, 2003, 24(3): 46-49. DOI: 10.7671/j.issn.1001-411X.2003.03.012
[9]	ZHONG Tan-wei. Numerical Simulation for Dynamic Melted Polymer Pipe Flows in Vibration Field[J]. Journal of South China Agricultural University, 2001, 22(1): 94-94. DOI: 10.7671/j.issn.1001-411X.2001.01.030
[10]	Huang Zhuolie. INDUCTION OF GLYCOLATE OXIDASE IN HIGHER PLANTS BY LIGHT[J]. Journal of South China Agricultural University, 1988, (3).

Cited By

Cited by

Periodical cited type(5)

1.	许文英. 西南桦的生长特征及栽培技术探讨. 农村科学实验. 2024(07): 82-84 .
2.	连晓倩，田云龙，黄健，钟兆全，周垂帆，黄伟，吴鹏飞. 杉木林分密度对套种闽楠树型、光合能力及生物量分配的影响. 林业科学研究. 2024(05): 13-22 .
3.	余清海. 不同基质·遮阴强度对盐肤木种子萌发的影响. 安徽农业科学. 2023(23): 90-92+103 .
4.	代嵩华，黄印冉，冯树香，闫淑芳，刘易超，陈丽英，樊彦聪. 不同遮阴强度对白桦幼苗生长情况的影响. 河北林业科技. 2022(02): 25-27+36 .
5.	朱秀征，靳超，房丽莎，刘震，王艳梅，蔡齐飞，李志，耿晓东. 不同密度下山桐子自然整枝差异性分析. 河南科学. 2022(08): 1257-1263 .

Other cited types(3)

Get Citation

PDF

XML

Article views (51) PDF downloads (18) Cited by(8)

Turn off MathJax

Article Contents

Abstract

References

3D reconstruction and semantic segmentation of fruit trees based on NeRF and improved RandLA-Net