数据质量多种性质的关联关系研究
作者:
基金项目:

国家重点基础研究发展计划(973)(2012CB316200);国家自然科学基金(U1509216,61472099,61133002);黑龙江省留学回国人员基金(LC2016026)


Association Relationships Study of Multi-Dimensional Data Quality
Author:
Fund Project:

National Program on Key Basic ResearchProject of China (973) (2012CB316200); National Natural Science Foundation of China (U1509216, 61472099, 61133002); Scientific Research Foundation for the Returned Overseas Chinese Scholars of Heilongjiang Provience (LC2016026)

  • 摘要
  • | |
  • 访问统计
  • |
  • 参考文献 [22]
  • |
  • 相似文献 [20]
  • |
  • 引证文献
  • | |
  • 文章评论
    摘要:

    信息化时代数据海量增长的同时,用户需要利用多种指标从不同性质角度对数据质量进行评价和改善.但在目前数据质量管理过程中,影响数据可用性的多种重要因素并非完全孤立,在评估机制和指导数据清洗规则时,彼此会发生关联.研究了在实际信息系统中适用的综合性数据质量评估方法,将文献所提出以及在实际的信息系统中常用的数据质量性质指标按其定义与性质进行了归纳总结,提出了基于性质的数据质量综合评估框架.之后针对影响数据可用性的4个重要性质:精确性、完整性、一致性以及时效性整理出在数据集合上的操作方法,并逐一介绍其违反模式的定义,随后给出其具体关系证明,进而确定数据质量多维关联关系评估策略,并通过实验验证了该策略的有效性.

    Abstract:

    Recently, with the rapid growth of data quantity, users are using a variety of indicators to evaluate and improve the quality of data from different dimensions. During the course of data quality management, it is found that many important factors that influence the data availability are not completely isolated. In the evaluation mechanism which can guide data cleaning rules, these dimensions may be associated with each other. In this paper, serveral data quality dimensions researched in the literature as well as being used in the real information system are discussed, and accordingly the definition and properties of the dimensions are summarized. In addition, a multi-dimensional data quality assessment framework is proposed. According to the four important properties of data availability:Accuracy, completeness, consistency and currency, the operation method and the relationships among them on the data set are constructed. Finally, a multi-dimensional data quality accessment strategy is created. The effctiveness of the proposed strategy is verified by experiments.

    参考文献
    [1] Mayer-Schonberger V, Cukier K. Big Data:A Revolution That Will Transform How We Live, Work, and Think. London:Houghton Mifflin Harcourt, 2013.19-31.
    [2] Sidi F, Shariat PPH, Affendey LS, Jabar MA, Ibrahim H, Mustapha A. Data quality:A survey of data quality dimensions. In:Proc. of the 2012 Int'l Conf. on Information Retrieval & Knowledge Management. IEEE, 2012.300-304.[doi:10.1109/InfRKM.2012.6204995]
    [3] Guo ZM, Zhou AY. Research on data quality and data cleaning:A survey. Ruan Jian Xue Bao/Journal of Software, 2002, 13(11):2076-2082(in Chinese with English abstract). http://www.jos.org.cn/ch/reader/view_abstract.aspx?flag=1&file_no=20021103&journal_id=jos
    [4] Batini C, Cappiello C, Francalanci C, Maurino A. Methodologies for data quality assessment and improvement. ACM Computing Surveys, 2009,41(3):No.16.[doi:10.1145/1541880.1541883]
    [5] Wang RY, Strong DM. Beyond accuracy:What data quality means to data consumers. Journal of Management Information Systems, 1996,12(4):5-33.[doi:10.1080/07421222.1996.11518099]
    [6] Cong G, Fan W, Geerts F, Jia XB, Ma S. Improving data quality:Consistency and accuracy. In:Proc. of the 33rd Int'l Conf. on Very Large Data Bases. VLDB Endowment, 2007.315-326. http://dl.acm.org/citation.cfm?id=1325890&preflayout=flat
    [7] Bohannon P, Fan W, Geerts F, Jia XB, Kementsietsidis A. Conditional functional dependencies for data cleaning. In:Proc. of the 23rd IEEE Int'l Conf. on Data Engineering. Istanbul:IEEE, 2007.746-755.[doi:10.1109/ICDE.2007.367920]
    [8] Fan W, Geerts F, Wijsen J. Determining the currency of data. ACM Trans. on Database Systems, 2012,37(4):25-41.[doi:10.1145/2389241.2389244]
    [9] Li MH, Li JZ, Gao H. Evaluation of data currency. Chinese Journal of Computers, 2012,35(11):2348-2360(in Chinese with English abstract).
    [10] McGilvray D. Executing Data Quality Projects:Ten Steps to Quality Data and Trusted Information. Burlington:Elsevier, 2008.16-59.
    [11] Fan W, Ma S, Tang N, Yu WY. Interaction between record matching and data repairing. Journal of Data and Information Quality, 2014,4(4):16.[doi:10.1145/2567657]
    [12] Tee SW, Bowen PL, Doyle PH. Rohde F. Factors influencing organizations to improve data quality in their information systems. Accounting & Finance, 2007,47(2):335-355.[doi:10.1111/j.1467-629x.2006.00205.x]
    [13] Eckerson W. Data quality and the bottom line, Vol.1. TDWI Report, Data Warehouse Institute, 2002.1-31.
    [14] Pipino LL, Lee YW, Wang RY. Data quality assessment. Communications of the ACM, 2002,45(4):211-218.[doi:10.1145/505248.506010]
    [15] https://en.wikipedia.org/wiki/Cronbach%27s_alpha
    [16] Yue K. Data Engineering:Processing, Analysis and Service. Beijing:Tsinghua University Press, 2013.169-180(in Chinese).
    [17] Fan W, Geerts F. Relative information completeness. ACM Trans. on Database Systems, 2010,35(4):97-106.[doi:10.1145/1862919.1862924]
    [18] Bravo L, Fan W, Ma S. Extending dependencies with conditions. In:Proc. of the 33rd Int'l Conf. on Very Large Data Bases. VLDB Endowment, 2007.243-254. http://dl.acm.org/citation.cfm?id=1325882&CFID=627672245&CFTOKEN=70772333
    附中文参考文献:
    [3] 郭志懋,周傲英.数据质量和数据清洗研究综述.软件学报,2002,13(11):2076-2082. http://www.jos.org.cn/ch/reader/view_abstract.aspx?flag=1&file_no=20021103&journal_id=jos
    [9] 李默涵,李建中,高宏.数据时效性判定问题的求解算法.计算机学报,2012,35(11):2348-2360.
    [16] 岳昆.数据工程——处理、分析与服务.北京:清华大学出版社,2013.169-180.
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

丁小欧,王宏志,张笑影,李建中,高宏.数据质量多种性质的关联关系研究.软件学报,2016,27(7):1626-1644

复制
分享
文章指标
  • 点击次数:6758
  • 下载次数: 7886
  • HTML阅读次数: 3141
  • 引用次数: 0
历史
  • 收稿日期:2015-10-10
  • 最后修改日期:2016-01-12
  • 在线发布日期: 2016-03-24
文章二维码
您是第19732356位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62562563 传真:010-62562533 Email:jos@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号