基于嵌入模型的知识图谱准确性评估
作者:
中图分类号:

TP182

基金项目:

国家自然科学基金(72201275)


Knowledge Graph Accuracy Evaluation Using Embedding Model
Author:
  • 摘要
  • | |
  • 访问统计
  • |
  • 参考文献 [35]
  • | |
  • 引证文献
  • | |
  • 文章评论
    摘要:

    知识图谱构造常面临三元组错误或缺失等质量问题, 准确性评估是选择和优化知识图谱的基础, 对提升下游应用的可信性具有重要意义. 引入嵌入模型, 降低对人工标注数据的依赖性, 提升大规模数据处理效率. 将三元组正误判定转化为无标注的自动化阈值选择问题, 提出了3种阈值选择策略, 增强评估的鲁棒性. 提出结合三元组重要性的评估方法, 从网络结构和关系语义两方面定义重要性指标, 对关键结构、频繁访问的三元组赋予更高关注度. 从嵌入模型表征能力、知识图谱稠密度、三元组重要性计算方式等多个角度, 分析比较了对评估方法性能的影响. 实验表明, 相比现有知识图谱准确性的自动化评估方法, 在零样本条件下, 所提出的方法可有效降低评估误差, 平均降低接近30%, 在错误率较高、稠密图谱的数据集上效果尤为显著.

    Abstract:

    Quality issues, such as errors or deficiencies in triplets, become increasingly prominent in knowledge graphs, severely affecting the credibility of downstream applications. Accuracy evaluation is crucial for building confidence in the use and optimization of knowledge graphs. An embedding-model-based method is proposed to reduce reliance on manually labeled data and to achieve scalable automatic evaluation. Triplet verification is formulated as an automated threshold selection problem, with three threshold selection strategies proposed to enhance the robustness of the evaluation. In addition, triplet importance indicators are incorporated to place greater emphasis on critical triplets, with importance scores defined based on network structure and relationship semantics. Experiments are conducted to analyze and compare the impact on performance from various perspectives, such as embedding model capacity, knowledge graph sparsity, and triplet importance definition. The results demonstrate that, compared to existing automated evaluation methods, the proposed method can significantly reduce evaluation errors by nearly 30% in zero-shot conditions, particularly on datasets of dense graphs with high error rates.

    参考文献
    [1] Suchanek FM, Kasneci G, Weikum G. YAGO: A core of semantic knowledge. In: Proc. of the 16th Int’l Conf. on World Wide Web. Banff: ACM, 2007. 697–706. [doi: 10.1145/1242572.1242667]
    [2] Lehmann J, Isele R, Jakob M, Jentzsch A, Kontokostas D, Mendes PN, Hellmann S, Morsey M, van Kleef P, Auer S, Bizer C. DBpedia—A large-scale, multilingual knowledge base extracted from Wikipedia. Semantic Web, 2015, 6(2): 167–195.
    [3] Carlson A, Betteridge J, Kisiel B, Settles B, Hruschka E, Mitchell T. Toward an architecture for never-ending language learning. In: Proc. of the 24th AAAI Conf. on Artificial Intelligence. Atlanta: AAAI, 2010. 1306–1313. [doi: 10.1609/aaai.v24i1.7519]
    [4] Bollacker K, Evans C, Paritosh P, Sturge T, Taylor J. Freebase: A collaboratively created graph database for structuring human knowledge. In: Proc. of the 2008 ACM SIGMOD Int’l Conf. on Management of Data. Vancouver: ACM, 2008. 1247–1250. [doi: 10.1145/1376616.1376746]
    [5] Ojha P, Talukdar P. KGEval: Accuracy estimation of automatically constructed knowledge graphs. In: Proc. of the 2017 Conf. on Empirical Methods in Natural Language Processing. Copenhagen: ACL, 2017. 1741–1750. [doi: 10.18653/v1/D17-1183]
    [6] Wang YQ, Ma FL, Gao J. Efficient knowledge graph validation via cross-graph representation learning. In: Proc. of the 29th ACM Int’l Conf. on Information & Knowledge Management. ACM, 2020. 1595–1604. [doi: 10.1145/3340531.3411902]
    [7] Gerber D, Esteves D, Lehmann J, Bühmann R, Usbeck R, Ngomo ACN, Speck R. DeFacto-temporal and multilingual deep fact validation. Journal of Web Semantics, 2015, 35: 85–101.
    [8] Liu SY, d’Aquin M, Motta E. Measuring accuracy of triples in knowledge graphs. In: Proc. of the 1st Int’l Conf. on Language, Data, and Knowledge. Galway: Springer, 2017. 343–357. [doi: 10.1007/978-3-319-59888-8_29]
    [9] Jia SB, Xiang Y, Chen XJ, Wang K, Shi J. Triple trustworthiness measurement for knowledge graph. In: Proc. of the 2019 World Wide Web Conf. San Francisco: ACM, 2019. 2865–2871. [doi: 10.1145/3308558.3313586]
    [10] Sedova A, Roth B. ACTC: Active threshold calibration for cold-start knowledge graph completion. In: Proc. of the 61st Annual Meeting of the Association for Computational Linguistics. Toronto: ACL, 2023. 1853–1863. [doi: 10.18653/v1/2023.acl-short.158]
    [11] Li Q, Li YL, Gao J, Su L, Zhao B, Demirbas M, Fan W, Han JW. A confidence-aware approach for truth discovery on long-tail data. Proc. of the VLDB Endowment, 2014, 8(4): 425–436.
    [12] Bordes A, Usunier N, Garcia-Duran A, Weston J, Yakhnenko O. Translating embeddings for modeling multi-relational data. In: Proc. of the 27th Int’l Conf. on Neural Information Processing Systems. Lake Tahoe: ACM, 2013. 2787–2795.
    [13] Trouillon T, Welbl J, Riedel S, Gaussier E, Bouchard G. Complex embeddings for simple link prediction. In: Proc. of the 33rd Int’l Conf. on Machine Learning. New York: PMLR, 2016. 2071–2080.
    [14] Schlichtkrull M, Kipf TN, Bloem P, van den Berg R, Titov I, Welling M. Modeling relational data with graph convolutional networks. In: Proc. of the 15th Int’l Conf. on the Semantic Web. Heraklion: Springer, 2018. 593–607. [doi: 10.1007/978-3-319-93417-4_38]
    [15] Sun ZQ, Deng ZH, Nie JY, Tang J. RotatE: Knowledge graph embedding by relational rotation in complex space. In: Proc. of the 7th Int’l Conf. on Learning Representations. New Orleans: OpenReview.net, 2019.
    [16] Xie RB, Liu ZY, Lin F, Lin LY. Does William Shakespeare REALLY write Hamlet? Knowledge representation learning with confidence. In: Proc. of the 32nd AAAI Conf. on Artificial Intelligence. New Orleans: AAAI, 2017. 4954–4961. [doi: 10.1609/aaai.v32i1.11924]
    [17] Speranskaya M, Schmitt M, Roth B. Ranking vs. Classifying: Measuring knowledge base completion quality. In: Proc. of the 2020 Conf. on Automated Knowledge Base Construction. AKBC, 2020. [doi: 10.24432/C57G65]
    [18] Wang Q, Mao ZD, Wang B, Guo L. Knowledge graph embedding: A survey of approaches and applications. IEEE Trans. on Knowledge and Data Engineering, 2017, 29(12): 2724–2743.
    [19] Park N, Kan A, Dong XL, Zhao T, Faloutsos C. Estimating node importance in knowledge graphs using graph neural networks. In: Proc. of the 25th ACM SIGKDD Int’l Conf. on Knowledge Discovery & Data Mining. Anchorage: ACM, 2019: 596–606. [doi: 10.1145/3292500.3330855]
    [20] Page L, Brin S, Motwani R, Winograd T. The PageRank citation ranking: Bringing order to the Web. 1999. http://ilpubs.stanford.edu:8090/422/
    [21] Li XT, Ng MK, Ye YM. HAR: Hub, authority and relevance scores in multi-relational data for query search. In: Proc. of the 12th SIAM Int’l Conf. on Data Mining. Anaheim: SIAM, 2012. 141–152. [doi: 10.1137/1.9781611972825]
    [22] Han X, Cao SL, Lv X, Lin YK, Liu ZY, Sun MS, Li JZ. OpenKE: An open toolkit for knowledge embedding. In: Proc. of the 2018 Conf. on Empirical Methods in Natural Language Processing: System Demonstrations. Brussels: ACL, 2018. 139–144. [doi: 10.18653/v1/D18-2024]
    [23] Zhang QG, Dong JN, Duan KY, Huang X, Liu YZ, Xu LC. Contrastive knowledge graph error detection. In: Proc. of the 31st ACM Int’l Conf. on Information & Knowledge Management. Atlanta: ACM, 2022. 2590–2599. [doi: 10.1145/3511808.3557264]
    [24] Ge CC, Gao YJ, Weng HH, Zhang C, Miao XY, Zheng BH. KGClean: An embedding powered knowledge graph cleaning framework. arXiv:2004.14478, 2020.
    [25] Akrami F, Saeef MS, Zhang QH, Hu W, Li CK. Realistic re-evaluation of knowledge graph completion methods: An experimental study. In: Proc. of the 2020 ACM SIGMOD Int’l Conf. on Management of Data. Portland: ACM, 2020. 1995–2010. [doi: 10.1145/3318464.3380599]
    [26] Gao JY, Li X, Xu YE, Sisman B, Dong XL, Yang J. Efficient knowledge graph accuracy evaluation. Proc. of the VLDB Endowment, 2019, 12(11): 1679–1691.
    [27] Pujara J, Augustine E, Getoor L. Sparsity and noise: Where knowledge graph embeddings fall short. In: Proc. of the 2017 Conf. on Empirical Methods in Natural Language Processing. Copenhagen: ACL, 2017. 1751–1756. [doi: 10.18653/v1/D17-1184]
    [28] Wang RY, Strong DM. Beyond accuracy: What data quality means to data consumers. Journal of Management Information Systems, 1996, 12(4): 5–33.
    [29] Zaveri A, Rula A, Maurino A, Pietrobon R, Lehmann J, Auer S. Quality assessment for linked data: A survey: A systematic literature review and conceptual framework. Semantic Web, 2016, 7(1): 63–93.
    [30] Xue BC, Zou L. Knowledge graph quality management: A comprehensive survey. IEEE Trans. on Knowledge and Data Engineering, 2023, 35(5): 4969–4988.
    [31] Acosta M, Zaveri A, Simperl E, Kontokostas D, Fl?ck F, Lehmann J. Detecting linked data quality issues via crowdsourcing: A DBpedia study. Semantic Web, 2016, 9(3): 303–335.
    [32] Qi YF, Zheng WG, Hong L, Zou L. Evaluating knowledge graph accuracy powered by optimized human-machine collaboration. In: Proc. of the 28th ACM SIGKDD Conf. on Knowledge Discovery and Data Mining. Washington: ACM, 2022. 1368–1378. [doi: 10.1145/3534678.3539233]
    [33] Amaral G, Rodrigues O, Simperl E. ProVe: A pipeline for automated provenance verification of knowledge graphs against textual sources. Semantic Web, 2024, 15(6): 2159–2192.
    [34] Fan WF. Dependencies for graphs: Challenges and opportunities. Journal of Data and Information Quality, 2019, 11(2): 1–12.
    [35] Fan WF, Hu CM, Liu XL, Lu P. Discovering graph functional dependencies. ACM Trans. on Database Systems (TODS), 2020, 45(3): 15.
    相似文献
    引证文献
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

张明韬,杨国利,白晓颖.基于嵌入模型的知识图谱准确性评估.软件学报,,():1-21

复制
分享
文章指标
  • 点击次数:42
  • 下载次数: 106
  • HTML阅读次数: 0
  • 引用次数: 0
历史
  • 收稿日期:2024-07-13
  • 最后修改日期:2024-09-26
  • 在线发布日期: 2025-06-18
文章二维码
您是第20262821位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62562563 传真:010-62562533 Email:jos@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号