Nighttime cattle face recognition based on cross-modal shared feature learning
-
摘要:目的
解决夜间环境下牛只身份信息难以有效识别的问题,以期为牛只全天候监测提供技术基础。
方法提出了一种基于跨模态共享特征学习的夜间牛脸识别方法。首先,模型框架采用浅层双流结构,有效提取不同模态的牛脸图像中的共享特征信息;其次,引入Triplet注意力机制,跨维度地捕捉交互信息,以增强牛只身份信息的提取;最后,通过嵌入扩展模块进一步挖掘跨模态身份信息的表征。
结果本文提出的夜间牛脸识别模型在测试集上的平均精度均值、一阶累积匹配特征值(CMC-1)和五阶累积匹配特征值(CMC-5)分别为90.68%、94.73%和97.82%,相较于未进行跨模态训练的模型,提高了19.67、18.91和12.00个百分点。
结论本研究所提出的模型为夜间牛只身份识别问题提供了可靠的解决方案,为实现牛只全天候持续监测奠定了坚实的技术基础。
Abstract:ObjectiveTo address the challenge of effectively recognizing cattle identity in the nighttime, and lay the technical foundation for 24-hour monitoring of cattle.
MethodA nighttime cattle face recognition method based on cross-modal shared feature learning was proposed. The model framework adopted a shallow dual-stream structure to effectively extract shared feature information from different modalities of cattle face images. Additionally, a triplet attention mechanism was introduced to capture intermodal interaction information across dimensions, enhancing the extraction of cattle identity information. Finally, an embedded extension module was utilized to further explore the representation of cross-modal identity information.
ResultThe nighttime cattle face recognition model proposed in this article achieved a mean average precision, the first order cumulative matching eigenvalue (CMC-1) and the fifth order cumulative matching eigenvalue (CMC-5) of 90.68%, 94.73% and 97.82% on the test set, respectively. Compared to the model without cross-modality training, the three indexes improved by 19.67, 18.91 and 12.00 percentage points, respectively.
ConclusionThe proposed method provides a reliable solution for nighttime cattle identity recognition, laying a solid technical foundation for the application of continuous 24-hour monitoring of cattle.
-
Keywords:
- Cattle /
- Identification /
- Heterogeneous face recognition /
- Cross-modality /
- Attention mechanism /
- Shared feature /
- Nighttime
-
-
图 2 夜间牛脸识别模型
嵌入空间中圆形和五边形色块分别表示原始嵌入和扩展嵌入,虚线和实线色块分别表示RGB图像和IR图像的嵌入,色块的不同颜色表示不同身份的牛只个体
Figure 2. Night cattle face recognition model
In the embedded space, circles and pentagons represent the original and extended embeddings respectively, dashed and solid color blocks represent the embeddings of RGB images and IR images respectively, different colors of the color blocks represent individual cattles of different identities
图 5 嵌入扩展模块
$ {\boldsymbol{f}} $代表原始的嵌入特征,$ {\boldsymbol{f}}_ + ^i $代表第i个分支生成的扩展嵌入特征,$ \theta _{3 \times 3}^n( \cdot ) $代表扩张率为n的3×3空洞卷积,$ {{{F}}_{{\mathrm{ReLU}}}}( \cdot ) $代表非线性激活函数,$ {\delta _{1 \times 1}}( \cdot ) $代表1×1卷积
Figure 5. Embedding expansion module
$ {\boldsymbol{f}} $ represents the original embedded features, $ {\boldsymbol{f}}_ + ^i $ represents the extended embedded features generated by the i-th branch, $ \theta _{3 \times 3}^n( \cdot ) $ represents a 3×3 dilated convolution with a dilation rate of n, $ {{F}_{{\mathrm{ReLU}}}}( \cdot ) $ represents a nonlinear activation function, and $ {\delta _{1 \times 1}}( \cdot ) $ represents a 1×1 convolution
表 1 RGB-IR跨模态牛脸识别数据集具体细节
Table 1 Overview of RGB-IR cross-modal cattle face recognition dataset
数据集
Dataset牛只数量
Number of cattlesRGB图像数量
Number of RGB imagesIR图像数量
Number of IR images图像总数量
Total number of images训练集 Training set 60 2570 2002 4572 测试集 Test set 32 1362 1085 2447 表 2 ResNet模型结构
Table 2 Model structure of ResNet
阶段
Stage操作
Operation重复次数
Stack number1 Conv, 7×7, 64, stride 2
Max pool, 3×3, stride 21 2 Conv, 1×1, 64
Conv, 3×3, 64
Conv, 1×1, 2563 3 Conv, 1×1, 128
Conv, 3×3, 128
Conv, 1×1, 1284 4 Conv, 1×1, 256
Conv, 3×3, 256
Conv, 1×1,1024 6 5 Conv, 1×1, 512
Conv, 3×3, 512
Conv, 1×1,2048 3 表 3 跨模态训练效果对比试验结果1)
Table 3 Comparative experimental result of cross-modal training effect
模型 Model mAP/% CMC-1/% CMC-5/% 未进行跨模态训练的模型 Model without cross-modal training 71.01 75.82 85.82 提出的模型 Proposed model 90.68 94.73 97.82 1) mAP:平均精度均值;CMC-1:一阶累积匹配特征值;CMC-5:五阶累积匹配特征值
1) mAP: Mean average precision; CMC-1: Cumulative match characteristic at rank 1; CMC-5:Cumulative match characteristic at rank 5表 4 各个模型消融试验结果1)
Table 4 Result of ablation experiment for each model
模型结构
Model structureTriplet注意力机制
Triplet attention
mechanism嵌入扩展模块
Embedding extension
modulesmAP/% CMC-1/% CMC-5/% Parameters/M FLOPs/G 单流
Single-stream77.23 84.55 91.64 9.18 4.73 √ 81.13 86.91 92.36 9.24 4.75 √ 79.00 85.09 92.36 9.18 4.73 √ √ 81.42 87.09 92.91 9.24 4.75 全双流
Full dual-stream80.73 88.73 94.00 9.18 4.73 √ 84.88 87.45 95.09 9.24 4.75 √ 82.75 89.27 93.64 9.18 4.73 √ √ 87.96 91.64 96.17 9.24 4.75 浅层双流
Shallow dual-stream81.86 89.45 94.73 9.18 4.73 √ 89.60 92.00 95.09 9.24 4.75 √ 86.19 89.64 96.55 9.18 4.73 √ √ 90.68 94.73 97.82 9.24 4.75 1) mAP:平均精度均值;CMC-1:一阶累积匹配特征值;CMC-5:五阶累积匹配特征值;Parameters:参数量;FLOPs:浮点运算量
1) mAP: Mean average precision; CMC-1: Cumulative match characteristic at rank 1; CMC-5:Cumulative match characteristic at rank 5; Parameters: Number of parameters; FLOPs: Floating point operations -
[1] 熊安然, 熊本海, 蒋林树. 奶牛数字化养殖技术研究进展[J]. 中国乳业, 2020, 11: 29-32. [2] 杨亮, 王辉, 陈睿鹏, 等. 畜禽个体身份标识技术发展进程与展望[J]. 猪业科学, 2023, 40(9): 24-27. doi: 10.3969/j.issn.1673-5358.2023.09.005 [3] KAUR A, KUMAR M, JINDAL M K. Cattle identification with muzzle pattern using computer vision technology: A critical review and prospective[J]. Soft Computing, 2022, 26(10): 4771-4795. doi: 10.1007/s00500-022-06935-x
[4] 许兴时, 王云飞, 华志新, 等. 融合YOLOv5s与通道剪枝算法的奶牛轻量化个体识别方法[J]. 农业工程学报, 2023, 39(15): 152-162. doi: 10.11975/j.issn.1002-6819.202303122 [5] LI R, WEN Y, ZHANG S, et al. Automated measurement of beef cattle body size via key point detection and monocular depth estimation[J]. Expert Systems with Applications, 2024, 244: 123042. doi: 10.1016/j.eswa.2023.123042
[6] XU X, WANG Y, SHANG Y, et al. Few-shot cow identification via meta-learning[J]. Information Processing in Agriculture, 2024, 4: 1-11.
[7] 王政, 宋怀波, 王云飞, 等. 奶牛运动行为智能监测研究进展与技术趋势[J]. 智慧农业, 2022, 4(2): 36-52. doi: 10.12133/j.smartag.SA202203011 [8] HOSSAIN M, KABIR M, ZHENG L, et al. A systematic review of machine learning techniques for cattle identification: Datasets, methods and future directions[J]. Artificial Intelligence in Agriculture, 2022, 6: 138-155. doi: 10.1016/j.aiia.2022.09.002
[9] MAHMUD M, ZAHID A, DAS A, et al. A systematic literature review on deep learning applications for precision cattle farming[J]. Computers and Electronics in Agriculture, 2021, 187: 106313. doi: 10.1016/j.compag.2021.106313
[10] QIAO Y, KONG H, CLARK C, et al. Intelligent perception for cattle monitoring: A review for cattle identification, body condition score evaluation, and weight estimation[J]. Computers and Electronics in Agriculture, 2021, 185: 106143. doi: 10.1016/j.compag.2021.106143
[11] HUANG X, HU Z, QIAO Y, et al. Deep learning-based cow tail detection and tracking for precision livestock farming[J]. IEEE/ASME Transactions on Mechatronics, 2023, 28(3): 1213-1221. doi: 10.1109/TMECH.2022.3175377
[12] FERREIRA R, BRESOLIN T, ROSA G, et al. Using dorsal surface for individual identification of dairy calves through 3D deep learning algorithms[J]. Computers and Electronics in Agriculture, 2022, 201: 107272. doi: 10.1016/j.compag.2022.107272
[13] WENG Z, MENG F, LIU S, et al. Cattle face recognition based on a two-branch convolutional neural network[J]. Computers and Electronics in Agriculture, 2022, 196: 106871. doi: 10.1016/j.compag.2022.106871
[14] LU Y, WENG Z, ZHENG Z, et al. Algorithm for cattle identification based on locating key area[J]. Expert Systems with Applications, 2023, 228: 120365. doi: 10.1016/j.eswa.2023.120365
[15] XU B, WANG W, GUO L, et al. CattleFaceNet: A cattle face identification approach based on RetinaFace and ArcFace loss[J]. Computers and Electronics in Agriculture, 2022, 193: 106675. doi: 10.1016/j.compag.2021.106675
[16] BAKHSHAYESHI I, ERFANI E, TAGHIKHAH F, et al. An intelligence cattle reidentification system over transport by siamese neural networks and YOLO[J]. IEEE Internet of Things Journal, 2024, 11(2): 2351-2363. doi: 10.1109/JIOT.2023.3294944
[17] YANG L, XU X, ZHAO J, et al. Fusion of RetinaFace and improved FaceNet for individual cow identification in natural scenes[J/OL]. Information Processing in Agriculture, (2023-09-02) [2024-05-01]. https://doi.org/10.1016/j.inpa.2023.09.001.
[18] SENGER P L. The estrus detection problem: New concepts, technologies, and possibilities[J]. Journal of Dairy Science, 1994, 77(9): 2745-2753. doi: 10.3168/jds.S0022-0302(94)77217-9
[19] WANG Z, HUA Z, WEN Y, et al. E-YOLO: Recognition of estrus cow based on improved YOLOv8n model[J]. Expert Systems with Applications, 2024, 238: 122212. doi: 10.1016/j.eswa.2023.122212
[20] CASEY T, PLAUT K. Circadian clocks and their integration with metabolic and reproductive systems: Our current understanding and its application to the management of dairy cows[J]. Journal of Animal Science, 2022, 100(10): 233.
[21] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2016: 770-778.
[22] MISRA D, NALAMADA T, ARASANIPALAI A U, et al. Rotate to attend: Convolutional triplet attention module[C]//2021 IEEE Winter Conference on Applications of Computer Vision. Waikoloa, HI, USA: IEEE, 2021: 3139-3148.
[23] ZHANG Y, WANG H. Diverse embedding expansion network and low-light cross-modality benchmark for visible-infrared person re-identification[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver, Canada: IEEE, 2023: 2153-2162.
[24] WANG Y, XU X, WANG Z, et al. ShuffleNet-Triplet: A lightweight RE-identification network for dairy cows in natural scenes[J]. Computers and Electronics in Agriculture, 2023, 205: 107632. doi: 10.1016/j.compag.2023.107632