Multimodal Data Fusion for Few-shot Named Entity Recognition Method
Author:
Affiliation:

  • Article
  • | |
  • Metrics
  • |
  • Reference [28]
  • |
  • Related [20]
  • | | |
  • Comments
    Abstract:

    As a crucial subtask in natural language processing (NLP), named entity recognition (NER) aims to extract the import information from text, which can help many downstream tasks such as machine translation, text generation, knowledge graph construction, and multi-modal data fusion to deeply understand the complex semantic information of the text and effectively complete these tasks. In practice, due to time and labor costs, NER suffers from annotated data scarcity, known as few-shot NER. Although few-shot NER methods based on text have achieved sound generalization performance, the semantic information that the model can extract is still limited due to the few samples, which leads to the poor prediction effect of the model. To this end, this study proposes a few-shot NER model based on the multi-modal dataset fusion, which provides additional semantic information with multi-modal data for the first time, to help the model prediction and can further effectively improve the effect of multimodal data fusion and modeling. This method converts image information into text information as auxiliary modality information, which effectively solves the problem of poor modality alignment caused by the inconsistent granularity of semantic information contained in text and images. In order to effectively consider the label dependencies in few-shot NER, this study uses the CRF framework and introduces the state-of-the-art meta-learning methods as the emission module and the transition module, respectively. To alleviate the negative impact of noisy samples in the auxiliary modal samples, this study proposes a general denoising network based on the idea of meta-learning. The denoising network can measure the variability of the samples and evaluate the beneficial extent of each sample to the model. Finally, this study conducts extensive experiments on real unimodal and multimodal data sets. The experimental results show the outstanding generalization performance of the proposed method, where the proposed method outperforms the state-of-the-art methods by 10 F1 scores in the 1-shot setting.

    Reference
    [1] Mikheev A, Moens M, Grover C. Named entity recognition without gazetteers. In:Proc. of the 9th Conf. of the European Chapter of the Association for Computational Linguistics. Stroudsburg:Association for Computational Linguistics, 1999. 1-8.
    [2] Fei H, Wu SQ, Li JY, et al. LasUIE:Unifying information extraction with latent adaptive structure-aware generative language model. In:Proc. of the 26th Conf. on Neural Information Processing Systems. 2022. 15460-15475.
    [3] Schäfer H, Idrissi-Yaghir A, Horn P, et al. Cross-language transfer of high-quality annotations:Combining neural machine translation with cross-linguistic span alignment to apply NER to clinical texts in a low-resource language. In:Proc. of the 4th Clinical Natural Language Processing Workshop. Stroudsburg:Association for Computational Linguistics, 2022. 53-62.
    [4] Bień M, Gilski M, Maciejewska M, et al. RecipeNLG:A cooking recipes dataset for semi-structured text generation. In:Proc. of the 13th Int'l Conf. on Natural Language Generation. Stroudsburg:Association for Computational Linguistics, 2020. 22-28.
    [5] Fan RY, Wang LZ, Yan JN, et al. Deep learning-based named entity recognition and knowledge graph construction for geological hazards. ISPRS Int'l Journal of Geo-information, 2019, 9(1):15-37.
    [6] Wu YZ, Li HR, Yao T, et al. A survey of multimodal information processing frontiers:Application, fusion and pre-training. Journal of Chinese Information Processing, 2022, 36(5):1-20(in Chinese with English abstract).
    [7] Sun L, Wang JQ, Zhang K, et al. RpBERT:A text-image relation propagation-based BERT model for multimodal NER. In:Proc. of the AAAI Conf. on Artificial Intelligence. Palo Alto:Association for the Advancement of Artificial Intelligence, 2021. 13860-13868.[doi:10.3969/j.issn.1003-0077.2022.05.001]
    [8] Liu W, Xu TG, Xu QH, et al. An encoding strategy based word-character LSTM for Chinese NER. In:Proc. of the 2019 Conf. of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. Stroudsburg:Association for Computational Linguistics, 2019. 2379-2389.[doi:10.18653/v1/N19-1247]
    [9] Jia C, Zhang Y. Multi-cell compositional LSTM for NER domain adaptation. In:Proc. of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg:Association for Computational Linguistics, 2020. 5906-5917.
    [10] Chang Y, Kong L, Jia K, et al. Chinese named entity recognition method based on BERT. In:Proc. of the 2021 IEEE Int'l Conf. on Data Science and Computer Application (ICDSCA). Piscataway:IEEE, 2021. 294-299.
    [11] Ju SG, Li TN, Sun JP. Chinese fine-grained name entity recognition based on associated memory networks. Ruan Jian Xue Bao/Journal of Software, 2021, 32(8):2545-2556(in Chinese with English abstract). http://www.jos.org.cn/1000-9825/6114.htm[doi:10.13328/j.cnki.jos.006114]
    [12] Zhao SJ. Named entity recognition in biomedical texts using an HMM model. In:Proc. of the Int'l Joint Workshop on Natural Language Processing in Biomedicine and Its Applications. Stroudsburg:Association for Computational Linguistics, 2004. 84-87.
    [13] Zhou GD, Su J. Named entity recognition using an HMM-based chunk tagger. In:Proc. of the 40th Annual Meeting of the Association for Computational Linguistics. Stroudsburg:Association for Computational Linguistics, 2002. 473-480.
    [14] Konkol M, Konopík M. CRF-based Czech named entity recognizer and consolidation of Czech NER research. In:Proc. of the 16th Int'l Conf. on Text, Speech, and Dialogue. Berlin:Springer, 2013. 153-160.
    [15] Liu J, Chen Y, Xu J. Low-resource NER by data augmentation with prompting. In:Proc. of the 31st Int'l Joint Conf. on Artificial Intelligence. 2022. 4252-4258.
    [16] Ma RT, Zhou X, Gui T, et al. Template-free prompt tuning for few-shot NER. In:Proc. of the 2022 Conf. of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. Stroudsburg:Association for Computational Linguistics, 2022. 5721-5732.[doi:10.18653/v1/2022.naacl-main.420]
    [17] Mettes P, van der Pol E, Snoek CGM. Hyperspherical prototype networks. In:Proc. of the 33rd Int'l Conf. on Neural Information Processing Systems. 2019. 1487-1497.
    [18] Li G, Jampani V, Sevilla-Lara L, et al. Adaptive prototype learning and allocation for few-shot segmentation. In:Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2021. 8334-8343.
    [19] Du PF, Li XY, Gao YL. Survey on multimodal visual language representation learning. Ruan Jian Xue Bao/Journal of Software, 2021, 32(2):327-348(in Chinese with English abstract). http://www.jos.org.cn/1000-9825/6125.htm[doi:10.13328/j.cnki.jos. 006125]
    [20] Yin J, Zhang ZD, Gao YH, Yang ZW, Li L, Xiao M, Sun YQ, Yan CG. Survey on vision-language pre-training. Ruan Jian Xue Bao/Journal of Software, 2023, 34(5):2000-2023(in Chinese with English abstract). http://www.jos.org.cn/1000-9825/6774.htm[doi:10.13328/j.cnki.jos.006774]
    [21] Chen SG, Aguilar G, Neves L, et al. Can images help recognize entities? A study of the role of images for Multimodal NER. In:Proc. of the 7th Workshop on Noisy User-generated Text (W-NUT 2021). Stroudsburg:Association for Computational Linguistics, 2021. 87-96.[doi:10.18653/v1/2021.wnut-1.11]
    [22] Wang XY, Gui M, Jiang Y, et al. ITA:Image-text alignments for multi-modal named entity recognition. In:Proc. of the 2022 Conf. of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. Stroudsburg:Association for Computational Linguistics, 2022. 3176-3189.
    [23] Hou YT, Che WX, Lai YK, et al. Few-shot slot tagging with collapsed dependency transfer and label- enhanced task-adaptive projection 据椀愀琀椀漀渀?昀漀爀?琀栀攀??搀瘀愀渀挀攀洀攀渀琀?漀昀??爀琀椀昀椀挀椀愀氀??渀琀攀氀氀椀最攀渀挀攀??? ??????????????嬀搀漀椀?? ??? ??愀愀愀椀?瘀??椀???????崀?戀爀?嬀? 崀??甀????一攀瘀攀猀?????愀爀瘀愀氀栀漀?嘀??攀琀?愀氀??嘀椀猀甀愀氀?愀琀琀攀渀琀椀漀渀?洀漀搀攀氀?昀漀爀?渀愀洀攀?琀愀最最椀渀最?椀渀?洀甀氀琀椀洀漀搀愀氀?猀漀挀椀愀氀?洀攀搀椀愀???渀?倀爀漀挀??漀昀?琀栀攀???琀栀??渀渀甀愀氀??攀攀琀椀渀最?漀昀?琀栀攀??猀猀漀挀椀愀琀椀漀渀?昀漀爀??漀洀瀀甀琀愀琀椀漀渀愀氀??椀渀最甀椀猀琀椀挀猀??匀琀爀漀甀搀猀戀甀爀最??猀猀漀挀椀愀琀椀漀渀?昀漀爀??漀洀瀀甀琀愀琀椀漀渀愀氀??椀渀最甀椀猀琀椀挀猀??? ??????? ???????戀爀?嬀??崀?夀甀??????椀愀渀最????夀愀渀最??椀??攀琀?愀氀???洀瀀爀漀瘀椀渀最?洀甀氀琀椀洀漀搀愀氀?渀愀洀攀搀?攀渀琀椀琀礀?爀攀挀漀最渀椀琀椀漀渀?瘀椀愀?攀渀琀椀琀礀?匀瀀愀渀?搀攀琀攀挀琀椀漀渀?眀椀琀栀?甀渀椀昀椀攀搀?洀甀氀琀椀洀漀搀愀氀?琀爀愀渀猀昀漀爀洀攀爀???渀?倀爀漀挀??漀昀?琀栀攀???琀栀??渀渀甀愀氀??攀攀琀椀渀最?漀昀?琀栀攀??猀猀漀挀椀愀琀椀漀渀?昀漀爀??漀洀瀀甀琀愀琀椀漀渀愀氀??椀渀最甀椀猀琀椀挀猀??匀琀爀漀甀搀猀戀甀爀最??猀猀漀挀椀愀琀椀漀渀?昀漀爀??漀洀瀀甀琀愀琀椀漀渀愀氀??椀渀最甀椀猀琀椀挀猀??? ? ????????????嬀搀漀椀?? ???????瘀??? ? ?愀挀氀?洀愀椀渀?? ?崀?戀爀?嬀??崀??愀渀最??夀??圀愀渀最?堀????攀渀最?娀儀??攀琀?愀氀????一一?刀???瘀愀爀椀愀琀椀漀渀愀氀?洀攀洀漀爀礀?愀甀最洀攀渀琀攀搀?洀漀搀攀氀?昀漀爀?挀爀漀猀猀?搀漀洀愀椀渀?昀攀眀?猀栀漀琀?渀愀洀攀搀?攀渀琀椀琀礀?爀攀挀漀最渀椀琀椀漀渀???渀?倀爀漀挀??漀昀?琀栀攀???猀琀??渀渀甀愀氀??攀攀琀椀渀最?漀昀?琀栀攀??猀猀漀挀椀愀琀椀漀渀?昀漀爀??漀洀瀀甀琀愀琀椀漀渀愀氀??椀渀最甀椀猀琀椀挀猀?嘀漀氀?????漀渀最?倀愀瀀攀爀猀???匀琀爀漀甀搀猀戀甀爀最??猀猀漀挀椀愀琀椀漀渀?昀漀爀??漀洀瀀甀琀愀琀椀漀渀愀氀??椀渀最甀椀猀琀椀挀猀??? ???????????????戀爀?嬀??崀??栀攀渀??圀???椀甀?儀???椀渀??夀??攀琀?愀氀???攀眀?猀栀漀琀?渀愀洀攀搀?攀渀琀椀琀礀?爀攀挀漀最渀椀琀椀漀渀?眀椀琀栀?猀攀氀昀?搀攀猀挀爀椀戀椀渀最?渀攀琀眀漀爀欀猀???渀?倀爀漀挀??漀昀?琀栀攀?? 琀栀??渀渀甀愀氀??攀攀琀椀渀最?漀昀?琀栀攀??猀猀漀挀椀愀琀椀漀渀?昀漀爀??漀洀瀀甀琀愀琀椀漀渀愀氀??椀渀最甀椀猀琀椀挀猀??匀琀爀漀甀搀猀戀甀爀最??猀猀漀挀椀愀琀椀漀渀?昀漀爀??漀洀瀀甀琀愀琀椀漀渀愀氀??椀渀最甀椀猀琀椀挀猀??? ???????????????戀爀?嬀??崀?嘀椀渀礀愀氀猀?伀???氀甀渀搀攀氀氀?????椀氀氀椀挀爀愀瀀?吀??攀琀?愀氀???愀琀挀栀椀渀最?渀攀琀眀漀爀欀猀?昀漀爀?漀渀攀?猀栀漀琀?氀攀愀爀渀椀渀最???渀?倀爀漀挀??漀昀?琀栀攀?? 琀栀??渀琀?氀??漀渀昀??漀渀?一攀甀爀愀氀??渀昀漀爀洀愀琀椀漀渀?倀爀漀挀攀猀猀椀渀最?匀礀猀琀攀洀猀??? ???????????????戀爀?嬀??崀??爀椀琀稀氀攀爀?????漀最愀挀栀攀瘀愀?嘀???爀攀琀漀瘀?????攀眀?猀栀漀琀?挀氀愀猀猀椀昀椀挀愀琀椀漀渀?椀渀?渀愀洀攀搀?攀渀琀椀琀礀?爀攀挀漀最渀椀琀椀漀渀?琀愀猀欀???渀?倀爀漀挀??漀昀?琀栀攀???琀栀?????匀???倀倀?匀礀洀瀀??漀渀??瀀瀀氀椀攀搀??漀洀瀀甀琀椀渀最??一攀眀?夀漀爀欀??猀猀漀挀椀愀琀椀漀渀?昀漀爀??漀洀瀀甀琀椀渀最??愀挀栀椀渀攀爀礀??? ?????????   ??戀爀?嬀??崀??愀?吀吀???椀愀渀最??儀??圀甀?儀???攀琀?愀氀???攀挀漀洀瀀漀猀攀搀?洀攀琀愀?氀攀愀爀渀椀渀最?昀漀爀?昀攀眀?猀栀漀琀?渀愀洀攀搀?攀渀琀椀琀礀?爀攀挀漀最渀椀琀椀漀渀???渀?倀爀漀挀??漀昀?琀栀攀??椀渀搀椀渀最猀?漀昀?琀栀攀??猀猀漀挀椀愀琀椀漀渀?昀漀爀??漀洀瀀甀琀愀琀椀漀渀愀氀??椀渀最甀椀猀琀椀挀猀??匀琀爀漀甀搀猀戀甀爀最??猀猀漀挀椀愀琀椀漀渀?昀漀爀??漀洀瀀甀琀愀琀椀漀渀愀氀??椀渀最甀椀猀琀椀挀猀??? ???????????????戀爀?嬀??崀?匀漀甀稀愀????一漀最甀攀椀爀愀?刀???漀琀甀昀漀?刀??倀漀爀琀甀最甀攀猀攀?渀愀洀攀搀?攀渀琀椀琀礀?爀攀挀漀最渀椀琀椀漀渀?甀猀椀渀最???刀吀??刀???愀爀堀椀瘀??? ??? ?????? ????戀爀?嬀??崀??攀瘀氀椀渀?????栀愀渀最??圀???攀攀????攀琀?愀氀????刀吀?倀爀攀?琀爀愀椀渀椀渀最?漀昀?搀攀攀瀀?戀椀搀椀爀攀挀琀椀漀渀愀氀?琀爀愀渀猀昀漀爀洀攀爀猀?昀漀爀?氀愀渀最甀愀最攀?甀渀搀攀爀猀琀愀渀搀椀渀最???渀?倀爀漀挀??漀昀?琀栀攀?一???????吀??匀琀爀漀甀搀猀戀甀爀最??猀猀漀挀椀愀琀椀漀渀?昀漀爀??漀洀瀀甀琀愀琀椀漀渀愀氀??椀渀最甀椀猀琀椀挀猀??? ??????????????嬀搀漀椀?? ???????瘀??一???????崀?戀爀?嬀??崀??椀攀搀攀爀椀欀?倀????椀洀洀礀?????搀愀洀???洀攀琀栀漀搀?昀漀爀?猀琀漀挀栀愀猀琀椀挀?漀瀀琀椀洀椀稀愀琀椀漀渀??愀爀堀椀瘀????????? ??? ????戀爀???蝎??螀???戀爀?嬀?崀??????一楧????????????ū潏???????鐀??贰?豔葔?????蝎潏晠???? ????????????? ??戀爀?嬀??崀??????一?腙??夀?獵??切蹗獎呑??兟?葾?蝎?鉾?絞?鹔卛????漀?晎???? ?????????????????????栀琀琀瀀???眀眀眀?樀漀猀?漀爀最?挀渀??   ???????????栀琀洀嬀搀漀椀?? ???????樀?挀渀欀椀?樀漀猀?  ????崀?戀爀?嬀??崀?尀佧???一???????????ū????梊膈晟恛?癸????漀?晎???? ???????????????????栀琀琀瀀???眀眀眀?樀漀猀?漀爀最?挀渀??   ???????????栀琀洀嬀搀漀椀?? ???????樀?挀渀欀椀?樀漀猀?  ????崀?戀爀?嬀? 崀???????????螚???栀穧蝦??一???阀銀??夀驛???鰀????????蒊??????漀?晎???? ???????????   ?? ????栀琀琀瀀???眀眀眀?樀漀猀?漀爀最?挀渀??   ???????????栀琀洀嬀搀漀椀?? ???????樀?挀渀欀椀?樀漀猀?  ????崀?ce. Palo Alto:Association for the Advancement of Artificial Intelligence, 2021. 13916-13923.[doi:10.1609/aaai.v35i15.17639]
    [35] Alayrac JB, Donahue J, Luc P, et al. Flamingo:A visual language model for few-shot learning. In:Proc. of the 36th Conf. on Neural Information Processing Systems. 2022. 23716-23736.
    [36] Ma X, Hovy E. End-to-end sequence labeling via Bi-directional LSTM-CNNs-CRF. In:Proc. of the 54th Annual Meeting of the Association for Computational Linguistics. Stroudsburg:Association for Computational Linguistics, 2016. 268-278.
    [37] Forney GD. The Viterbi algorithm. Proc. of the IEEE, 61(3):268-278.[doi:10.1109/PROC.1973.9030]
    [38] Ding N, Xu GW, Chen YL, et al. Few-NERD:A few-shot named entity recognition dataset. In:Proc. of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int'l Joint Conf. on Natural Language Processing, Vol. 1(Long Papers). Stroudsburg:Association for Computational Linguistics, 2021. 3198-3213.
    [39] Zhang Q, Fu JL, Liu XY, et al. Adaptive co-attention network for named entity recognition in Tweets. In:Proc. of the 32nd AAAI Conf. on Artificial Intelligence. Palo Alto:Asso????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
    Cited by
    Comments
    Comments
    分享到微博
    Submit
Get Citation

张天明,张杉,刘曦,曹斌,范菁.融合多模态数据的小样本命名实体识别方法.软件学报,2024,35(3):1107-1124

Copy
Share
Article Metrics
  • Abstract:1200
  • PDF: 4457
  • HTML: 2903
  • Cited by: 0
History
  • Received:July 15,2023
  • Revised:September 05,2023
  • Online: November 08,2023
  • Published: March 06,2024
You are the first2044067Visitors
Copyright: Institute of Software, Chinese Academy of Sciences Beijing ICP No. 05046678-4
Address:4# South Fourth Street, Zhong Guan Cun, Beijing 100190,Postal Code:100190
Phone:010-62562563 Fax:010-62562533 Email:jos@iscas.ac.cn
Technical Support:Beijing Qinyun Technology Development Co., Ltd.

Beijing Public Network Security No. 11040202500063