融合任务知识的多模态知识图谱补全
作者:
作者单位:

作者简介:

通讯作者:

中图分类号:

TP18

基金项目:

国家自然科学基金(62206193, 62076176, 62076175)


Task Knowledge Fusion for Multimodal Knowledge Graph Completion
Author:
Affiliation:

Fund Project:

  • 摘要
  • |
  • 图/表
  • |
  • 访问统计
  • |
  • 参考文献
  • |
  • 相似文献
  • |
  • 引证文献
  • |
  • 资源附件
  • |
  • 文章评论
    摘要:

    知识图谱补全任务旨在根据已有的事实三元组(头实体、关系、尾实体)来挖掘知识图谱中缺失的事实三元组. 现有的研究工作主要致力于利用知识图谱中的结构信息来进行知识图谱补全任务. 然而, 这些工作忽略了知识图谱中蕴含的其他模态的信息也可能对知识图谱补全有帮助. 并且, 由于基于特定任务的知识通常没有被注入通用的预训练模型, 因而如何在抽取模态信息的过程中融合任务的相关知识变得至关重要. 此外, 因为不同模态特征的对于知识图谱补全的贡献不一样, 所以如何有效地保留有用的多模态信息也是一大挑战. 为了解决上述问题, 本文提出了一种融合任务知识的多模态知识图谱补全方法. 我们利用在当前任务上微调过的多模态编码器, 来获取不同模态下的实体向量表示. 并且, 通过一个基于循环神经网络的模态融合过滤模块, 去除与任务无关的多模态特征. 最后, 利用同构图网络表征并更新所有特征, 从而有效地完成多模态知识图谱补全任务. 实验结果表明, 我们提出的方法能有效地抽取不同模态的信息, 并且能够通过进一步的多模态过滤融合来增强实体的表征能力, 进而提高多模态知识图谱补全任务的性能.

    Abstract:

    The task of completing knowledge graphs aims to reveal the missing fact triples within the knowledge graph based on existing fact triples (head entity, relation, tail entity). Existing research primarily focuses on utilizing the structural information within the knowledge graph. However, these efforts overlook that other modal information contained within the knowledge graph may also be helpful for knowledge graph completion. In addition, since task-specific knowledge is typically not integrated into general pre-training models, the process of incorporating task-related knowledge into modal information extraction becomes crucial. Moreover, given that different modal features contribute uniquely to knowledge graph completion, effectively preserving useful multimodal information poses a significant challenge. To address these issues, this paper proposes a multimodal knowledge graph completion method that incorporates task knowledge. It utilizes a fine-tuned multimodal encoder tailored to the current task to acquire entity vector representations across different modalities. Subsequently, a modal fusion-filtering module based on recurrent neural networks is utilized to eliminate task-independent multimodal features. Finally, the study utilizes a simple isomorphic graph network to represent and update all features, thus effectively accomplishing multimodal knowledge graph completion. Experimental results demonstrate the effectiveness of our approach in extracting information from different modalities. Furthermore, it shows that our method enhances entity representation capability through additional multimodal filtering and fusion, consequently improving the performance of multimodal knowledge graph completion tasks.

    参考文献
    相似文献
    引证文献
引用本文

陈强,张栋,李寿山,周国栋.融合任务知识的多模态知识图谱补全.软件学报,,():1-15

复制
分享
文章指标
  • 点击次数:
  • 下载次数:
  • HTML阅读次数:
  • 引用次数:
历史
  • 收稿日期:2023-08-25
  • 最后修改日期:2023-11-03
  • 录用日期:
  • 在线发布日期: 2024-07-03
  • 出版日期:
您是第位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62562563 传真:010-62562533 Email:jos@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号