• 《中国科学引文数据库(CSCD)》来源期刊
  • 中国科技期刊引证报告(核心版)期刊
  • 《中文核心期刊要目总览》核心期刊
  • RCCSE中国核心学术期刊

基于跨模态共享特征学习的夜间牛脸识别方法

许兴时, 王云飞, 邓红兴, 宋怀波

许兴时, 王云飞, 邓红兴, 等. 基于跨模态共享特征学习的夜间牛脸识别方法[J]. 华南农业大学学报, 2024, 45(5): 793-801. DOI: 10.7671/j.issn.1001-411X.202403020
引用本文: 许兴时, 王云飞, 邓红兴, 等. 基于跨模态共享特征学习的夜间牛脸识别方法[J]. 华南农业大学学报, 2024, 45(5): 793-801. DOI: 10.7671/j.issn.1001-411X.202403020
XU Xingshi, WANG Yunfei, DENG Hongxing, et al. Nighttime cattle face recognition based on cross-modal shared feature learning[J]. Journal of South China Agricultural University, 2024, 45(5): 793-801. DOI: 10.7671/j.issn.1001-411X.202403020
Citation: XU Xingshi, WANG Yunfei, DENG Hongxing, et al. Nighttime cattle face recognition based on cross-modal shared feature learning[J]. Journal of South China Agricultural University, 2024, 45(5): 793-801. DOI: 10.7671/j.issn.1001-411X.202403020

基于跨模态共享特征学习的夜间牛脸识别方法

基金项目: 国家重点研发计划(2023YFD1301800);国家自然科学基金(32272931);陕西省农业重点核心技术项目(2023NYGG005);陕西省科技创新引导计划(2022QFY11-02)
详细信息
    作者简介:

    许兴时,硕士研究生,主要从事模式识别研究,E-mail: xingshixu@nwafu.edu.cn

    通讯作者:

    宋怀波,教授,博士,主要从事精准养殖研究,E-mail: songhuaibo@nwsuaf.edu.cn

  • 中图分类号: TP391.4;S823

Nighttime cattle face recognition based on cross-modal shared feature learning

  • 摘要:
    目的 

    解决夜间环境下牛只身份信息难以有效识别的问题,以期为牛只全天候监测提供技术基础。

    方法 

    提出了一种基于跨模态共享特征学习的夜间牛脸识别方法。首先,模型框架采用浅层双流结构,有效提取不同模态的牛脸图像中的共享特征信息;其次,引入Triplet注意力机制,跨维度地捕捉交互信息,以增强牛只身份信息的提取;最后,通过嵌入扩展模块进一步挖掘跨模态身份信息的表征。

    结果 

    本文提出的夜间牛脸识别模型在测试集上的平均精度均值、一阶累积匹配特征值(CMC-1)和五阶累积匹配特征值(CMC-5)分别为90.68%、94.73%和97.82%,相较于未进行跨模态训练的模型,提高了19.67、18.91和12.00个百分点。

    结论 

    本研究所提出的模型为夜间牛只身份识别问题提供了可靠的解决方案,为实现牛只全天候持续监测奠定了坚实的技术基础。

    Abstract:
    Objective 

    To address the challenge of effectively recognizing cattle identity in the nighttime, and lay the technical foundation for 24-hour monitoring of cattle.

    Method 

    A nighttime cattle face recognition method based on cross-modal shared feature learning was proposed. The model framework adopted a shallow dual-stream structure to effectively extract shared feature information from different modalities of cattle face images. Additionally, a triplet attention mechanism was introduced to capture intermodal interaction information across dimensions, enhancing the extraction of cattle identity information. Finally, an embedded extension module was utilized to further explore the representation of cross-modal identity information.

    Result 

    The nighttime cattle face recognition model proposed in this article achieved a mean average precision, the first order cumulative matching eigenvalue (CMC-1) and the fifth order cumulative matching eigenvalue (CMC-5) of 90.68%, 94.73% and 97.82% on the test set, respectively. Compared to the model without cross-modality training, the three indexes improved by 19.67, 18.91 and 12.00 percentage points, respectively.

    Conclusion 

    The proposed method provides a reliable solution for nighttime cattle identity recognition, laying a solid technical foundation for the application of continuous 24-hour monitoring of cattle.

  • 图  1   数据集中的部分图像样本

    Figure  1.   Partial image samples in dataset

    图  2   夜间牛脸识别模型

    嵌入空间中圆形和五边形色块分别表示原始嵌入和扩展嵌入,虚线和实线色块分别表示RGB图像和IR图像的嵌入,色块的不同颜色表示不同身份的牛只个体

    Figure  2.   Night cattle face recognition model

    In the embedded space, circles and pentagons represent the original and extended embeddings respectively, dashed and solid color blocks represent the embeddings of RGB images and IR images respectively, different colors of the color blocks represent individual cattles of different identities

    图  3   Triplet注意力原理

    Figure  3.   Triplet attention schematic

    图  4   Triplet注意力结构

    Figure  4.   Triplet attention architecture

    图  5   嵌入扩展模块

    $ {\boldsymbol{f}} $代表原始的嵌入特征,$ {\boldsymbol{f}}_ + ^i $代表第i个分支生成的扩展嵌入特征,$ \theta _{3 \times 3}^n( \cdot ) $代表扩张率为n的3×3空洞卷积,$ {{{F}}_{{\mathrm{ReLU}}}}( \cdot ) $代表非线性激活函数,$ {\delta _{1 \times 1}}( \cdot ) $代表1×1卷积

    Figure  5.   Embedding expansion module

    $ {\boldsymbol{f}} $ represents the original embedded features, $ {\boldsymbol{f}}_ + ^i $ represents the extended embedded features generated by the i-th branch, $ \theta _{3 \times 3}^n( \cdot ) $ represents a 3×3 dilated convolution with a dilation rate of n, $ {{F}_{{\mathrm{ReLU}}}}( \cdot ) $ represents a nonlinear activation function, and $ {\delta _{1 \times 1}}( \cdot ) $ represents a 1×1 convolution

    图  6   训练过程参数变化曲线

    Figure  6.   Change curve of parameter in training process

    表  1   RGB-IR跨模态牛脸识别数据集具体细节

    Table  1   Overview of RGB-IR cross-modal cattle face recognition dataset

    数据集
    Dataset
    牛只数量
    Number of cattles
    RGB图像数量
    Number of RGB images
    IR图像数量
    Number of IR images
    图像总数量
    Total number of images
    训练集 Training set 60 2570 2002 4572
    测试集 Test set 32 1362 1085 2447
    下载: 导出CSV

    表  2   ResNet模型结构

    Table  2   Model structure of ResNet

    阶段
    Stage
    操作
    Operation
    重复次数
    Stack number
    1 Conv, 7×7, 64, stride 2
    Max pool, 3×3, stride 2
    1
    2 Conv, 1×1, 64
    Conv, 3×3, 64
    Conv, 1×1, 256
    3
    3 Conv, 1×1, 128
    Conv, 3×3, 128
    Conv, 1×1, 128
    4
    4 Conv, 1×1, 256
    Conv, 3×3, 256
    Conv, 1×1, 1024
    6
    5 Conv, 1×1, 512
    Conv, 3×3, 512
    Conv, 1×1, 2048
    3
    下载: 导出CSV

    表  3   跨模态训练效果对比试验结果1)

    Table  3   Comparative experimental result of cross-modal training effect

    模型 Model mAP/% CMC-1/% CMC-5/%
    未进行跨模态训练的模型 Model without cross-modal training 71.01 75.82 85.82
    提出的模型 Proposed model 90.68 94.73 97.82
     1) mAP:平均精度均值;CMC-1:一阶累积匹配特征值;CMC-5:五阶累积匹配特征值
     1) mAP: Mean average precision; CMC-1: Cumulative match characteristic at rank 1; CMC-5:Cumulative match characteristic at rank 5
    下载: 导出CSV

    表  4   各个模型消融试验结果1)

    Table  4   Result of ablation experiment for each model

    模型结构
    Model structure
    Triplet注意力机制
    Triplet attention
    mechanism
    嵌入扩展模块
    Embedding extension
    modules
    mAP/% CMC-1/% CMC-5/% Parameters/M FLOPs/G
    单流
    Single-stream
    77.23 84.55 91.64 9.18 4.73
    81.13 86.91 92.36 9.24 4.75
    79.00 85.09 92.36 9.18 4.73
    81.42 87.09 92.91 9.24 4.75
    全双流
    Full dual-stream
    80.73 88.73 94.00 9.18 4.73
    84.88 87.45 95.09 9.24 4.75
    82.75 89.27 93.64 9.18 4.73
    87.96 91.64 96.17 9.24 4.75
    浅层双流
    Shallow dual-stream
    81.86 89.45 94.73 9.18 4.73
    89.60 92.00 95.09 9.24 4.75
    86.19 89.64 96.55 9.18 4.73
    90.68 94.73 97.82 9.24 4.75
     1) mAP:平均精度均值;CMC-1:一阶累积匹配特征值;CMC-5:五阶累积匹配特征值;Parameters:参数量;FLOPs:浮点运算量
     1) mAP: Mean average precision; CMC-1: Cumulative match characteristic at rank 1; CMC-5:Cumulative match characteristic at rank 5; Parameters: Number of parameters; FLOPs: Floating point operations
    下载: 导出CSV
  • [1] 熊安然, 熊本海, 蒋林树. 奶牛数字化养殖技术研究进展[J]. 中国乳业, 2020, 11: 29-32.
    [2] 杨亮, 王辉, 陈睿鹏, 等. 畜禽个体身份标识技术发展进程与展望[J]. 猪业科学, 2023, 40(9): 24-27. doi: 10.3969/j.issn.1673-5358.2023.09.005
    [3]

    KAUR A, KUMAR M, JINDAL M K. Cattle identification with muzzle pattern using computer vision technology: A critical review and prospective[J]. Soft Computing, 2022, 26(10): 4771-4795. doi: 10.1007/s00500-022-06935-x

    [4] 许兴时, 王云飞, 华志新, 等. 融合YOLOv5s与通道剪枝算法的奶牛轻量化个体识别方法[J]. 农业工程学报, 2023, 39(15): 152-162. doi: 10.11975/j.issn.1002-6819.202303122
    [5]

    LI R, WEN Y, ZHANG S, et al. Automated measurement of beef cattle body size via key point detection and monocular depth estimation[J]. Expert Systems with Applications, 2024, 244: 123042. doi: 10.1016/j.eswa.2023.123042

    [6]

    XU X, WANG Y, SHANG Y, et al. Few-shot cow identification via meta-learning[J]. Information Processing in Agriculture, 2024, 4: 1-11.

    [7] 王政, 宋怀波, 王云飞, 等. 奶牛运动行为智能监测研究进展与技术趋势[J]. 智慧农业, 2022, 4(2): 36-52. doi: 10.12133/j.smartag.SA202203011
    [8]

    HOSSAIN M, KABIR M, ZHENG L, et al. A systematic review of machine learning techniques for cattle identification: Datasets, methods and future directions[J]. Artificial Intelligence in Agriculture, 2022, 6: 138-155. doi: 10.1016/j.aiia.2022.09.002

    [9]

    MAHMUD M, ZAHID A, DAS A, et al. A systematic literature review on deep learning applications for precision cattle farming[J]. Computers and Electronics in Agriculture, 2021, 187: 106313. doi: 10.1016/j.compag.2021.106313

    [10]

    QIAO Y, KONG H, CLARK C, et al. Intelligent perception for cattle monitoring: A review for cattle identification, body condition score evaluation, and weight estimation[J]. Computers and Electronics in Agriculture, 2021, 185: 106143. doi: 10.1016/j.compag.2021.106143

    [11]

    HUANG X, HU Z, QIAO Y, et al. Deep learning-based cow tail detection and tracking for precision livestock farming[J]. IEEE/ASME Transactions on Mechatronics, 2023, 28(3): 1213-1221. doi: 10.1109/TMECH.2022.3175377

    [12]

    FERREIRA R, BRESOLIN T, ROSA G, et al. Using dorsal surface for individual identification of dairy calves through 3D deep learning algorithms[J]. Computers and Electronics in Agriculture, 2022, 201: 107272. doi: 10.1016/j.compag.2022.107272

    [13]

    WENG Z, MENG F, LIU S, et al. Cattle face recognition based on a two-branch convolutional neural network[J]. Computers and Electronics in Agriculture, 2022, 196: 106871. doi: 10.1016/j.compag.2022.106871

    [14]

    LU Y, WENG Z, ZHENG Z, et al. Algorithm for cattle identification based on locating key area[J]. Expert Systems with Applications, 2023, 228: 120365. doi: 10.1016/j.eswa.2023.120365

    [15]

    XU B, WANG W, GUO L, et al. CattleFaceNet: A cattle face identification approach based on RetinaFace and ArcFace loss[J]. Computers and Electronics in Agriculture, 2022, 193: 106675. doi: 10.1016/j.compag.2021.106675

    [16]

    BAKHSHAYESHI I, ERFANI E, TAGHIKHAH F, et al. An intelligence cattle reidentification system over transport by siamese neural networks and YOLO[J]. IEEE Internet of Things Journal, 2024, 11(2): 2351-2363. doi: 10.1109/JIOT.2023.3294944

    [17]

    YANG L, XU X, ZHAO J, et al. Fusion of RetinaFace and improved FaceNet for individual cow identification in natural scenes[J/OL]. Information Processing in Agriculture, (2023-09-02) [2024-05-01]. https://doi.org/10.1016/j.inpa.2023.09.001.

    [18]

    SENGER P L. The estrus detection problem: New concepts, technologies, and possibilities[J]. Journal of Dairy Science, 1994, 77(9): 2745-2753. doi: 10.3168/jds.S0022-0302(94)77217-9

    [19]

    WANG Z, HUA Z, WEN Y, et al. E-YOLO: Recognition of estrus cow based on improved YOLOv8n model[J]. Expert Systems with Applications, 2024, 238: 122212. doi: 10.1016/j.eswa.2023.122212

    [20]

    CASEY T, PLAUT K. Circadian clocks and their integration with metabolic and reproductive systems: Our current understanding and its application to the management of dairy cows[J]. Journal of Animal Science, 2022, 100(10): 233.

    [21]

    HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2016: 770-778.

    [22]

    MISRA D, NALAMADA T, ARASANIPALAI A U, et al. Rotate to attend: Convolutional triplet attention module[C]//2021 IEEE Winter Conference on Applications of Computer Vision. Waikoloa, HI, USA: IEEE, 2021: 3139-3148.

    [23]

    ZHANG Y, WANG H. Diverse embedding expansion network and low-light cross-modality benchmark for visible-infrared person re-identification[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver, Canada: IEEE, 2023: 2153-2162.

    [24]

    WANG Y, XU X, WANG Z, et al. ShuffleNet-Triplet: A lightweight RE-identification network for dairy cows in natural scenes[J]. Computers and Electronics in Agriculture, 2023, 205: 107632. doi: 10.1016/j.compag.2023.107632

图(6)  /  表(4)
计量
  • 文章访问数:  573
  • HTML全文浏览量:  28
  • PDF下载量:  32
  • 被引次数: 0
出版历程
  • 收稿日期:  2024-05-09
  • 网络出版日期:  2024-06-26
  • 发布日期:  2024-07-14
  • 刊出日期:  2024-08-07

目录

    /

    返回文章
    返回