基于语义分割的芒果表皮缺陷识别

刘小刚; 李荣梅; 范诚; 杨启良; 赵璐

doi:10.7671/j.issn.1001-411X.202204014

基于语义分割的芒果表皮缺陷识别

刘小刚^1,,
李荣梅¹,
范诚^2, ,,
杨启良¹,
赵璐³

1.
昆明理工大学现代农业工程学院, 云南昆明 650000
2.
中联重科智能技术有限公司,湖南长沙 410013
3.
四川大学水利水电学院, 四川成都 610065

基金项目: 国家自然科学基金(51979133)；四川省科技计划(2022YFQ0082)

详细信息

作者简介:
刘小刚，教授，博士，主要从事农业工程相关研究，E-mail: liuxiaogangjy@126.com

通讯作者:
范　诚，工程师，主要从事工程机械视觉技术研发，E-mail: 17771848474@189.cn

中图分类号: S436.67；TP391
计量
- 文章访问数: 131
- HTML全文浏览量: 19
- PDF下载量: 289
出版历程
- 收稿日期: 2022-04-21
- 网络出版日期: 2023-05-17
- 刊出日期: 2023-01-09

Recognition of mango skin defect based on semantic segmentation

1.
Faculty of Modern Agricultural Engineering, Kunming University of Science and Technology, Kunming 650000, China
2.
Zoomlion Intelligent Technology Limited Company, Changsha 410013, China
3.
College of Water Resources and Hydropower, Sichuan University, Chengdu 610065, China

摘要

摘要:
目的
运用语义分割技术自动识别芒果及其表皮缺陷，实现芒果的质量评估及分选，为芒果质量快速无损检测提供参考。

方法
采集自然环境下的多场景芒果表皮缺陷图像用于模型的训练与测试，将联合上采样金字塔(Joint pyramid upsampling，JPU)结构替换DeepLabV3+中空洞空间卷积池化金字塔(Atrous spatial pyramid pooling，ASPP)，将Atrous-ResNet模型替换DeepLabV3+中Xception模型，采用类别像素准确率(Class pixel accuracy，CPA)、平均像素准确率(Mean pixel accuracy，MPA)、平均交并比 (Mean intersection over union，MIoU)作为模型的精度评价指标。

结果
采用JPU模块替换ASPP模块，在ResNet网络中运用扩张卷积有利于增大模型的感受野，总体上预测的边界更加平滑，且对细小缺陷的识别更精确；与SegNet、LinkNet算法的对比验证表明，Atrous-ResNet模型具备更高的精度，CPA小幅提升，MPA提升3.79个百分点，MIoU提升4.57个百分点，Atrous-ResNet模型具有更好的识别效果。

结论
基于语义分割的方法应用于芒果表皮缺陷识别是可行的，Atrous-ResNet模型较SegNet以及LinkNet算法比较具有更高的识别精度。
- 语义分割 /
- DeepLabV3+ /
- 联合上采样金字塔 /
- 缺陷检测 /
- 芒果表皮
Abstract:
Objective
The semantic segmentation technology was used to automatically identify mango and its skin defects, to realize the quality evaluation and sorting of mango and provide a reference for the rapid and nondestructive testing of mango quality.

Method
Mango skin defect images in multi-scene of natural environment were collected for model training and testing. Atrous spatial pyramid pooling (ASPP) in DeepLabV3+ was replaced by joint pyramid upsampling (JPU) structure, and Xception model in DeepLabV3+ was replaced by Atrous-ResNet model. Class pixel accuracy (CPA), mean pixel accuracy (MPA) and mean intersection over union (MIoU) were used as the accuracy evaluation indexes of each model.

Result
JPU module was used to replace ASPP module, and Atrous convolution was applied to ResNet network which was conductive to increase the receptive field of the model. In general, the predicted boundary was smoother, and the identification of small defects was more accurate. The comparison with SegNet and LinkNet algorithms showed that Atrous-ResNet model had higher accuracy, with CPA slightly improved, MPA was up 3.79 percent point and MIoU was up 4.57 percent. Atrous-ResNet model had better identification effects.

Conclusion
The method based on semantic segmentation is feasible for mango skin defect recognition. Compared with SegNet and LinkNet algorithms, Atrous-ResNet model has higher recognition accuracy.
- Semantic segmentation /
- DeepLabV3+ /
- Joint pyramid upsampling /
- Defect recognition /
- Mango skin

HTML全文

图 1 图像形态学处理

Figure 1. Image morphological processing

下载: 全尺寸图片幻灯片

图 2 联合上采样金字塔结构

a表示3个特征图作为输入参数；b表示上采样后的特征图并行经过不同膨胀率和分离卷积后再融合；c表示经过卷积生成最终的特征图

Figure 2. The structure of joint pyramid upsampling

a represents the three feature maps as input parameters, b represents the upsampled feature maps undergoing parallel operations of different expansion rates and separate convolution followed by concat, c represents the final feature map generated by convolution

下载: 全尺寸图片幻灯片

图 3 改进的算法架构图

Figure 3. Structure chart of improved algorithm

下载: 全尺寸图片幻灯片

图 4 不同模型评价指标的比较

Figure 4. Comparison of evaluation indexes for different models

下载: 全尺寸图片幻灯片

图 5 不同场景下的模型预测效果

Figure 5. Prediction performance in multi-scenarios

下载: 全尺寸图片幻灯片

图 6 LinkNet、SegNet及Atrous-ResNet模型在不同场景下的识别效果对比

Figure 6. Comparison of recognition performance among LinkNe, SegNet and Atrous-ResNet model in multi-scenarios

下载: 全尺寸图片幻灯片

表 1 DeepLabV3+与Atrous-ResNet模型的评价指标比较

Table 1 Comparison of evaluation indexes between DeepLabV3+ and Atrous-ResNet model %

模型 Model	类别像素准确率 Class pixel accuracy			平均像素准确率 Mean pixel accuracy	平均交并比 Mean intersection over union
模型 Model	芒果 Mango	缺陷 Defect	茎梗 Stalk	平均像素准确率 Mean pixel accuracy	平均交并比 Mean intersection over union
DeepLabV3+	92.03	91.29	88.74	90.69	89.56
Atrous-ResNet	94.16	95.32	93.95	94.48	94.13

下载: 导出CSV

表 2 不同算法的评价指标的比较

Table 2 Comparison of evaluation indexes of different algorithms %

模型 Model	类别像素准确率 Class pixel accuracy			平均像素准确率 Mean pixel accuracy	平均交并比 Mean intersection over union
模型 Model	芒果 Mango	缺陷 Defect	茎梗 Stalk	平均像素准确率 Mean pixel accuracy	平均交并比 Mean intersection over union
LinkNet	82.72	81.17	78.94	80.94	77.37
SegNet	80.49	77.14	75.68	77.77	72.69
Atrous-ResNet	94.27	94.58	93.03	93.96	92.65

下载: 导出CSV

参考文献(28)

[1]	DEEPA M, PUSHPA B, UDAYKUMAR K. Physicochemical properties, nutritional and antinutritional composition of pulp and peel of three mango varieties[J]. International Journal of Educational Science and Research, 2017, 7(3): 81-94.
[2]	KANGD, WANGY, FANY, et al. Research and development of Camellia oleifera fruit sheller and sorting machine[J]. Earth and Environmental Science, 2018, 108: 042051. doi: 10.1088/1755-1315/108/4/042051.
[3]	SIHOMBING P, TOMMY F, SEMBIRING S, et al. The Citrus fruit sorting device automatically based on color method by using tcs320 color sensor and arduino uno microcontroller[J]. Journal of Physics: Conference Series, 2019, 1235: 012064. doi: 10.1088/1742-6596/1235/1/012064.
[4]	邓继忠, 任高生, 兰玉彬, 等. 基于可见光波段的无人机超低空遥感图像处理[J]. 华南农业大学学报, 2016, 37(6): 16-22. doi: 10.7671/j.issn.1001-411X.2016.06.003
[5]	戴泽翰, 郑正, 黄莉舒, 等. 基于深度卷积神经网络的柑橘黄龙病症状识别[J]. 华南农业大学学报, 2020, 41(4): 111-119. doi: 10.7671/j.issn.1001-411X.201909031
[6]	赵德安, 吴任迪, 刘晓洋, 等. 基于YOLO深度卷积神经网络的复杂背景下机器人采摘苹果定位[J]. 农业工程学报, 2019, 35(3): 164-173. doi: 10.11975/j.issn.1002-6819.2019.03.021
[7]	刘小刚, 范诚, 李加念, 等. 基于卷积神经网络的草莓识别方法[J]. 农业机械学报, 2020, 51(2): 237-244. doi: 10.6041/j.issn.1000-1298.2020.02.026
[8]	XING S L, LEE M. Classification accuracy improvement for small-size Citrus pests and diseases using bridge connections in deep neural networks[J]. Sensors, 2020, 20(17): 4992. doi: 10.3390/s20174992.
[9]	SELVARAJ M G, VERGARA A, RUIZ H, et al. AI-powered banana diseases and pest detection[J]. Plant Methods, 2019, 15(1): 92. doi: 10.1186/s13007-018-0385-5
[10]	PATEL K K, KAR A, KHAN M A. Common external defect detection of mangoes using color computer vision[J]. Journal of the Institution of Engineers: Series A, 2019, 100(4): 559-568. doi: 10.1007/s40030-019-00396-6
[11]	刘平, 朱衍俊, 张同勋, 等. 自然环境下贴叠葡萄串的识别与图像分割算法[J]. 农业工程学报, 2020, 36(6): 161-169. doi: 10.11975/j.issn.1002-6819.2020.06.019
[12]	李江波, 彭彦昆, 黄文倩, 等. 桃子表面缺陷分水岭分割方法研究[J]. 农业机械学报, 2014, 45(8): 288-293. doi: 10.6041/j.issn.1000-1298.2014.08.046
[13]	张德军, 周学成, 杨旭东. 基于图像处理和深度迁移学习的芒果果实病状识别[J]. 华南农业大学学报, 2021, 42(4): 113-124. doi: 10.7671/j.issn.1001-411X.202011002
[14]	袁培森, 黎薇, 任守纲, 等. 基于卷积神经网络的菊花花型和品种识别[J]. 农业工程学报, 2018, 34(5): 152-158. doi: 10.11975/j.issn.1002-6819.2018.05.020
[15]	尚增强, 杨东福, 马质璞. 基于深度卷积神经网络的大豆叶片多种病害分类识别[J]. 大豆科学, 2021, 40(5): 662-668.
[16]	程曦, 吴云志, 张友华, 等. 基于深度卷积神经网络的储粮害虫图像识别[J]. 中国农学通报, 2018, 34(1): 154-158. doi: 10.11924/j.issn.1000-6850.casb16110146
[17]	JHURIA M, KUMAR A, BORSE R. Image processing for smart farming: Detection of disease and fruit grading[C]//IEEE 20^th International Conference on Image Information Processing. Shimla, India: IEEE, 2013: 521-526.
[18]	BASAVARAJ T, BHAVANA S. Banana plant disease detection and grading using image processing[J]. International Journal of Engineering Science and Computing, 2016: 6512-6516.
[19]	SAHU D, POTDAR M. Defect identification and maturity detection of mango fruits using image analysis[J]. American Journal of Artificial Intelligence, 2017, 1(1): 5-14.
[20]	HUANG X Y, LÜ R Q, WANG S, et al. Integration of computer vision and colorimetric sensor array for nondestructive detection of mango quality[J]. Journal of Food Process Engineering, 2018, 41(8): 1-9.
[21]	SUDARJAT, KUSUMIYATI, HASANUDDIN, et al. Rapid and non-destructive detection of insect infestations on intact mango by means of near infrared spectroscopy[J]. IPO Conference Series: Earth and Environmental Science, 2019, 365(1): 012037. doi: 10.1088/1755-1315/365/1/012037.
[22]	KESTUR R, MEDURI A, NARASIPURA O. MangoNet: A deep semantic segmentation architecture for a method to detect and count mangoes in an open orchard[J]. Engineering Applications of Artificial Intelligence, 2018, 77: 59-69.
[23]	SHELHAMER E, LONG J, DARRELL T. Fully convolutional networks for semantic segmentation[J]. Proceedings of the IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(4): 640-651. doi: 10.1109/TPAMI.2016.2572683
[24]	CHOLLET F. Xception: Deep learning with depthwise separable convolutions[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE, 2017: 1800-1807
[25]	WU H K, ZHANG J G, HUANG K Q, et al. FastFCN: Rethinking dilated convolution in the backbone for semantic segmentation[EB/OL]. ArXiv preprint arXiv, 2019: 1903.11816. (2019-03-28)[2022-03-20]. https://ariv.org/abs/1903.11816.
[26]	CHEN L C, ZHU Y, PAPANDREOU G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[M]//Computer Vision ECCV. Cham: Springer, 2018: 833-851
[27]	CHAURASIA A, CULURCIELLO E. LinkNet: Exploiting encoder representations for efficient semantic segmentation[J]. IEEE Visual Communications and Image Processing (VCIP), 2017: 1-4. doi: 10.1109/VCIP.2017.8305148.
[28]	BADRINARAYANAN V, KENDALL A, CIPOLLA R. SegNet: A deep convolutional encoder-decoder architecture for image segmentation[J]. Proceedings of the IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12): 2481-2495. doi: 10.1109/TPAMI.2016.2644615