数据标注研究综述
作者:
作者简介:

蔡莉(1975-),女,云南昆明人,博士,副教授,主要研究领域为数据质量,数据挖掘,智能交通;刘俊晖(1980-),男,博士,讲师,主要研究领域为模型驱动开发,计算机视觉;王淑婷(1994-),女,硕士生,主要研究领域为数据质量,数据挖掘;朱扬勇(1963-),男,博士,教授,博士生导师,CCF高级会员,主要研究领域为数据科学.

通讯作者:

蔡莉,E-mail:caili@ynu.edu.cn

基金项目:

国家自然科学基金(61663047,U1636207);云南大学服务云南行动计划(2016ZD05)


Survey of Data Annotation
Author:
Fund Project:

National Natural Science Foundation of China (61663047, U1636207); Project of Yunnan University Serves Yunnan Initiatives (2016ZD05)

  • 摘要
  • | |
  • 访问统计
  • |
  • 参考文献 [125]
  • |
  • 相似文献 [20]
  • | | |
  • 文章评论
    摘要:

    数据标注是大部分人工智能算法得以有效运行的关键环节.数据标注越准确、标注的数据量越大,算法的性能就越好.数据标注行业的发展带动了中国许多城市和城镇的就业,促使中国逐渐成为世界数据标注的中心.阐述了数据标注的发展概况,包括起源、应用场景、分类和任务;列举了目前常用的标注数据集、开源的数据标注工具和商业数据标注平台;提出了标注中的角色、标准和流程等数据标注规范;给出了一个情感分析场景中的数据标注实例;描述各类主流的标注质量评估算法及其特点,并对比它们优缺点;最后,从任务、工具、数据标注质量和安全性这4个方面对数据标注的研究方向和发展趋势进行了展望.

    Abstract:

    Data annotation is a key part of the effective operation of most artificial intelligence algorithms. The better the annotation accuracy and quantity, the better the performance of the algorithm. The development of the data annotation industry boosts employment in many cities and towns in China, prompting China to gradually become the center of world data annotation. This study summarizes its development, including origin, application scenarios, classifications, and tasks; lists the commonly used annotation data sets, open source data annotation tools and commercial annotation platforms; proposes the data annotation specification including roles, standards, and processes; gives an example of data annotation in a sentiment analysis. Then, this paper describes the models and characteristics of state-of-the-art algorithms for evaluating annotation results, and compares their advantages and disadvantages. Finally, this paper prospects research focuses and development trends of data annotation from four aspects:tasks, tools, annotation quality, and security.

    参考文献
    [1] Xuan Z. Hidden "foxconn" labor-intensive industry in artificial intelligence industry. Internet Weekly, 2018,675(21):28-29(in Chinese).
    [2] Bengio Y. Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2009,2(1):1-127.
    [3] Corea F. How AI is changing the insurance landscape. In:Applied Artificial Intelligence:Where AI Can Be Used in Business. 2019. 5-10.
    [4] Alonso O. Challenges with label quality for supervised learning. Journal of Data and Information Quality, 2015,6(1):1-3.
    [5] Brendel W, Rauber J, Bethge M, et al. Decision-based adversarial attacks:Reliable attacks against black-box machine learning models. In:Proc. of the Int'l Conf. on Learning Representations. 2018. 1-12.
    [6] IMAGENET. 2010. http://image-net.org/about-stats
    [7] Kornblith S, Shlens J, Le QV, et al. Do better ImageNet models transfer better? In:Proc. of the Computer Vision and Pattern Recognition. 2019. 2661-2671.
    [8] Zhu J, Kaplan R, Johnson J, et al. HiDDeN:Hiding data with deep networks. In:Proc. of the European Conf. on Computer Vision. 2018. 682-697.
    [9] Wang C, Blei DM, Li F, et al. Simultaneous image classification and annotation. In:Proc. of the Computer Vision and Pattern Recognition. 2009. 1903-1910.
    [10] Bearman A, Russakovsky O, Ferrari V, et al. What's the point:Semantic segmentation with point supervision. In:Proc. of the European Conf. on Computer Vision. 2016. 549-565.
    [11] Debattista J, Auer S, Lange C, et al. Luzzu-A methodology and framework for linked data quality assessment. Journal of Data and Information Quality, 2016,8(1):1-32.
    [12] Reitan EH, Saib SH. Computer graphics in an automatic aircraft landing system. In:Proc. of the National Computer Conf. 1976. 689-700.
    [13] Kodali RK, Jain V, Bose S, et al. IoT based smart security and home automation system. In:Proc. of the Int'l Conf. on Computing Communication and Automation. 2016. 1286-1289.
    [14] Syed L, Jabeen S, Manimala S, et al. Data science algorithms and techniques for smart healthcare using IoT and big data analytics:Towards smarter algorithms. In:Proc. of the Smart Techniques for a Smarter Planet. 2019. 211-241.
    [15] Khan M, Wu X, Xu X, et al. Big data challenges and opportunities in the hype of Industry 4.0. In:Proc. of the IEEE ICC 2017. IEEE, 2017. https://ieeexplore.ieee.org/document/7996801
    [16] Appen. 1996. https://appen.com/blog/how-ai-driving-innovation-ecommerce-retail/
    [17] Zhang Y, Lu Y. Research on the problems and strategies of rural E-commerce in the age of internet + agriculture. In:Proc. of the Semantics Knowledge and Grid. 2018. 257-260.
    [18] Christen P, Gayler RW, Tran K, et al. Automatic discovery of abnormal values in large textual databases. Journal of Data and Information Quality, 2016,7(1):1-31.
    [19] Sivarajah U, Kamal MM, Irani Z, et al. Critical analysis of big data challenges and analytical methods. Journal of Business Research, 2017,70:263-286.
    [20] Guo XM, Ma LL, Su K, et al. Research on automatic evaluation method of metadata quality of data repositories based on sematic annotation. Computer Applications and Software, 2018,35(6):29-33, 88(in Chinese with English abstract).
    [21] Khashabi D, Khot T, Sabharwal A, Clark P, Etzioni O, Roth D. Question answering via integer programming over semi-structured knowledge. In:Proc. of the Int'l Joint Conf. on Artificial Intelligence. 2016. 1145-1152.
    [22] Ling H, Gao J, Kar A, et al. Fast interactive object annotation with Curve-GCN. Computer Vision and Pattern Recognition, 2019, 1:5257-5266.
    [23] Barbosa L, Carvalho BW, Zadrozny B, et al. Pooling hybrid representations for Web structured data annotation. arXiv:1610. 00493, 2016.
    [24] Jing K. Review of evaluation of social tagging system. Jiangsu Science & Technology Information, 2018,35(11):8-10(in Chinese with English abstract).
    [25] Zhang L, Wang T, Liu Y, et al. A semi-structured information semantic annotation method for Web pages. Neural Computing and Applications, 2019,(5):1-11.
    [26] Cai L, Liang Y, Zhu YY, et al. History and development tendency of data quality. Computer Science, 2018,45(4):1-10(in Chinese with English abstract).
    [27] Egorow O, Lotz A, Siegert I, et al. Accelerating manual annotation of filled pauses by automatic pre-selection. In:Proc. of the Int'l Conf. on Companion Technology. 2018. 263-286.
    [28] Zheng G, Mukherjee S, Dong XL, et al. OpenTag:Open attribute value extraction from product profiles. In:Proc. of the Knowledge Discovery and Data Mining. 2018. 1049-1058.
    [29] Barthelmess P, Kaiser EC, Huang X, et al. Collaborative multimodal photo annotation over digital paper. In:Proc. of the Int'l Conf. on Multimodal Interfaces. 2006. 4-11.
    [30] Zhang B, Hao J, Ma G, et al. Automatic image annotation based on semi-paired probabilistic canonical correlation analysis. Ruan Jian Xue Bao/Journal of Software, 2017,28(2):292-309(in Chinese with English abstract). http://www.jos.org.cn/1000-98255047.htm[doi:10.13328/j.cnki.jos.005047]
    [31] Vielhauer C, Schott M, Kratzer C, et al. Nested object watermarking:Transparency and capacity evaluation. Electronic Imaging, 2008,6819:1-18.
    [32] Luo L, Yang Z, Yang P, et al. An attention-based BiLSTM-CRF approach to document-level chemical named entity recognition. Bioinformatics, 2018,34(8):1381-1388.
    [33] Pearson J, Robinson S, Jones M, et al. PaperChains:Dynamic sketch+voice annotations. In:Proc. of the Conf. on Computer Supported Cooperative Work. 2015. 383-392.
    [34] Ceolin D, Groth PT, Maccatrozzo V, et al. Combining user reputation and provenance analysis for trust assessment. Journal of Data and Information Quality, 2016,7(1):1-28.
    [35] Levinson J, Askeland J, Becker J, et al. Towards fully autonomous driving:Systems and algorithms. In:Proc. of the 2011 IEEE Intelligent Vehicles Symp. 2011. 163-168.
    [36] Hillier LW, Graves T, Fulton RS, et al. Generation and annotation of the DNA sequences of human chromosomes 2 and 4. Nature, 2005,434(7034):724-731.
    [37] Gambhir M, Gupta V. Recent automatic text summarization techniques:A survey. Artificial Intelligence Review, 2017,47(1):1-66.
    [38] Liu C, Su T, Yu L, et al. Self-correction method for automatic data annotation. In:Proc. of the Asian Conf. on Pattern Recognition. 2017. 911-916.
    [39] Uijlings JR, Konyushkova K, Lampert CH, et al. Learning intelligent dialogs for bounding box annotation. In:Proc. of the Computer Vision and Pattern Recognition. 2018. 9175-9184.
    [40] Wang C. Image annotation refinement using random walk with restarts. In:Proc. of the ACM Multimedia. 2016. 647-650.
    [41] Parmar BR, Jarrett TR, Burgon NS, et al. Comparison of left atrial area marked ablated in electroanatomical maps with scar in MRI. Journal of Cardiovascular Electrophysiology, 2014,25(5):457-463.
    [42] Perrott DR, Marlborough K. Minimum audible movement angle:Marking the end points of the path traveled by a moving sound source. The Journal of the Acoustical Society of America, 1989,85(4):1773-1775.
    [43] Best-Rowden L, Jain AK. Longitudinal study of automatic face recognition. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2017,40(1):148-162.
    [44] Müller RH, Clegg DL. Automatic paper chromatography. Annals of the New York Academy of Sciences, 1949,53(5):1108-1118.
    [45] Sikos LF. RDF-powered semantic video annotation tools with concept mapping to linked data for next-generation video indexing:A comprehensive review. Multimedia Tools and Applications, 2017,76(12):14437-14460.
    [46] Willis CG, Law E, Williams AC, et al. CrowdCurio:An online crowdsourcing platform to facilitate climate change studies using herbarium specimens. New Phytologist, 2017,215(1):479-488.
    [47] Dolmaya JM. Analyzing the crowdsourcing model and its impact on public perceptions of translation. Translator, 2012,18(2):167-191.
    [48] Aktas A, Alexa C, Andreev V, et al. Measurement of inclusive jet production in deep-inelastic scattering at high and determination of the strong coupling. Physics Letters B, 2007,653:134-144.
    [49] Chen K, Chang C, Wu C, et al. Quadrant of euphoria:A crowdsourcing platform for QoE assessment. IEEE Network, 2010,24(2):28-35.
    [50] Kohler T. Crowdsourcing-based business models:How to create and capture value. California Management Review, 2015,57(4):63-84.
    [51] Chalam KV, Jain P, Shah VA, et al. Evaluation of Web-based annotation of ophthalmic images for multicentric clinical trials. Indian Journal of Ophthalmology, 2006,54(2):479-488.
    [52] Tang J, Li H, Qi G, et al. Image annotation by graph-based inference with integrated multiple/single instance representations. IEEE Trans. on Multimedia, 2010,12(2):131-141.
    [53] Zhou H, Gao B, Wu J, et al. Adaptive feeding:Achieving fast and accurate detections by adaptively combining object detectors. Computer Vision and Pattern Recognition, 2017,1:3523-3533.
    [54] Jongejan B. Automatic annotation of head velocity and acceleration in Anvil. In:Proc. of the Language Resources and Evaluation. 2012. 201-208.
    [55] Tulasi RL, Rao MS, Ankita K, et al. Ontology-based automatic annotation:An approach for efficient retrieval of semantic results of Web documents. In:Proc. of the 1st Int'l Conf. on Computational Intelligence and Informatics. 2017. 57-63.
    [56] Xie C, Mao X, Huang J, et al. KOBAS 2.0:A Web server for annotation and identification of enriched pathways and diseases. In:Proc. of the Nucleic Acids Research. 2011. 316-322.
    [57] Liu P, Zhang Y. Data Annotation Engineering. Beijing:Tsinghua University Press, 2019(in Chinese).
    [58] Berriel RF, Rossi FS, De Souza AF, et al. Automatic large-scale data acquisition via crowdsourcing for crosswalk classification:A deep learning approach. In:Proc. of the Computers and Graphics. 2017. 32-42.
    [59] Boselli R, Cesarini M, Mercorio F, et al. An AI planning system for data cleaning. In:Proc. of the European Conf. on Machine Learning. 2017. 349-353.
    [60] Li R, Lin Z, Lin HL, et al. Text emotion analysis:A survey. Journal of Computer Research and Development, 2018,55(1):30-52(in Chinese with English abstract).
    [61] Lei LY. Research on fine-grained sentiment analysis base on Chinese[MS. Thesis]. Hengyang:University of South China, 2014(in Chinese with English abstract).
    [62] Cai L, Pan J, Wei BL, et al. Visualization analysis for spatio-temporal pattern of hotspots and sentiment change towards microblog check-in data. Miniature Microcomputer System, 2018,39(9):1889-1894(in Chinese with English abstract).
    [63] Cao W. Research of the algorithm of region-value annotation in crowdsouring[MS. Thesis]. Nanjing:Nanjing University of Finance and Economics, 2017(in Chinese with English abstract).
    [64] Gennari R, Tonelli S, Vittorini P, et al. Challenges in quality of temporal data-Starting with gold standards. Journal of Data and Information Quality, 2015,6(2):2-3.
    [65] Xu TZ, Xu ZY. Combination method of Fisher theory and the majority of voting for data fusion. Science and Technology Information, 2009,27:445,488(in Chinese with English abstract).
    [66] Wang Y, Rao Y, Zhan X, et al. Sentiment and emotion classification over noisy labels. In:Proc. of the Knowledge Based Systems. 2016. 207-216.
    [67] Snow R, O'Connor B, Jurafsky D, et al. Cheap and fast-But is it good? Evaluating non-expert annotations for natural language tasks. In:Proc. of the Conf. on Empirical Methods in Natural Language Processing. 2008. 254-263.
    [68] Sorokin A, Forsyth D. Utility data annotation with Amazon Mechanical Turk. In:Proc. of the 2008 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition Workshop on Internet Vision, 2008. 1-8.
    [69] Gupta R. Modeling multiple time series annotations as noisy distortions of the ground truth:An expectation-Maximization approach. IEEE Trans. on Affective Computing, 2018,9(1):76-89.
    [70] Zeng J, Liu Z, Cao X, et al. Fast online EM for big topic modeling. IEEE Trans. on Knowledge and Data Engineering, 2016,28(3):675-688.
    [71] Wang W, Zhou Z. Crowdsourcing label quality:A theoretical analysis. Science in China Series F:Information Sciences, 2015, 58(11):1-12.
    [72] Dawid AP. Skene AM. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society, 1979,28(1):20-28.
    [73] Raykar VC, Yu S, Zhao LH, et al. Supervised learning from multiple experts:Whom to trust when everyone lies a bit. In:Proc. of the 26th Int'l Conf. on Machine Learning. Quebec, 2009. 889-896.
    [74] Raykar VC, Yu S, Zhao LH, et al. Learning from crowds. Journal of Machine Learning Research, 2010,11(2):1297-1322.
    [75] Yu H, Chen Y. Clustering ensemble method using three-way decisions based on spark. Journal of Zhengzhou University (Natura Science Edition), 2018,50(1):23-29(in Chinese with English abstract).
    [76] Vogel T, Heise A, Draisbach U, et al. Reach for gold:An annealing standard to evaluate duplicate detection results. Journal of Data and Information Quality, 2014,5(1):1-25.
    [77] Papineni K, Roukos S, Ward T, et al. Bleu:A method for automatic evaluation of machine translation. In:Proc. of the Meeting of the Association for Computational Linguistics. 2002. 311-318.
    [78] Lavie A, Agarwal A. METEOR:An automatic metric for MT evaluation with high levels of correlation with human judgments. In:Proc. of the Workshop on Statistical Machine Translation. 2007. 228-231.
    [79] Jiang YH, Ding L, Li JE, et al. Abstractive summarization model considering hybrid lexical features. Journal of Hebei University of Science and Technology, 2019,40(2):152-158(in Chinese with English abstract).
    [80] Jin F, Huang M, Lu Z, et al. Towards automatic generation of gene summary. In:Proc. of the North American Chapter of the Association for Computational Linguistics. 2009. 97-105.
    [81] Plaza L. A semantic graph-based approach to biomedical summarization. Artificial Intelligence in Medicine, 2011,53(1):1-14.
    [82] Campr M, Ježek K. Comparing semantic models for evaluating automatic document summarization. In:Proc. of the Text Speech and Dialogue. 2015. 252-260.
    [83] Kang SZ, Hong MA, Huang RY. An opinion and MRW based sentiment summarization framework. Acta Electronica Sinica, 2017, 45(12):3005-3011.
    [84] Lin C. ROUGE:A package for automatic evaluation of summaries. In:Proc. of the Meeting of the Association for Computational Linguistics. 2004. 74-81.
    [85] Vedantam R, Zitnick CL, Parikh D, et al. CIDEr:Consensus-based image description evaluation. In:Proc. of the Computer Vision and Pattern Recognition. 2015. 4566-4575.
    [86] Chen T, Liao Y, Chuang C, et al. Show, adapt and tell:Adversarial training of cross-domain image captioner. arXiv:1705.00930, 2017.
    [87] Anderson P, Fernando B, Johnson M, et al. SPICE:Semantic propositional image caption evaluation. In:Proc. of the European Conf. on Computer Vision. 2016. 382-398.
    [88] Cui Y, Yang G, Veit A, et al. Learning to evaluate image captioning. In:Proc. of the Computer Vision and Pattern Recognition. 2018. 5804-5812.
    [89] Demartini G, Difallah DE, Cudremauroux P. ZenCrowd:Leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In:Proc. of the Int'l Conf. on World Wide Web. ACM, 2012. 469-478.
    [90] Zhang J, Sheng VS, Wu J, et al. Multi-class ground truth inference in crowdsourcing with clustering. IEEE Trans. on Knowledge and Data Engineering, 2016,28(4):1080-1085.
    [91] Ruckhaus E, Vidal M, Castillo S, et al. Analyzing linked data quality with LiQuate. In:Proc. of the European Semantic Web Conf. 2014. 488-493.
    [92] Ruiz N, Federico M. Phonetically-oriented word error alignment for speech recognition error analysis in speech translation. In:Proc. of the Automatic Speech Recognition and Understanding. 2016. 296-302.
    [93] Augello A, Cuzzocrea A, Pilato G, et al. An innovative similarity measure for sentence plagiarism detection. In:Proc. of the Int'l Conf. on Computational Science and Its Applications. 2016. 552-566.
    [94] Spiccia C, Augello A, Pilato G, et al. Semantic word error rate for sentence similarity. In:Proc. of the IEEE Int'l Conf. on Semantic Computing. 2016. 266-269.
    [95] Escudero JP, Novoa J, Mahu R, et al. An improved DNN-based spectral feature mapping that removes noise and reverberation for robust automatic speech recognition. arXiv:1803.09016, 2018.
    [96] Andriluka M, Uijlings JR, Ferrari V, et al. Fluid annotation:A human-machine collaboration interface for full image annotation. In:Proc. of the ACM Multimedia. 2018. 1957-1966.
    [97] Yang B, Kaul M, Jensen CS. Using incomplete information for complete weight annotation of road networks. IEEE Trans. on Knowledge and Data Engineering, 2014,26(5):1267-1279.
    [98] Marcheggiani D, Sebastiani F. On the effects of low-quality training data on information extraction from clinical reports. Journal of Data and Information Quality, 2017,9(1):1-25.
    [99] Maccartney B, Galley M, Manning CD, et al. A phrase-based alignment model for natural language inference. In:Proc. of the Empirical Methods in Natural Language Processing. 2008. 802-811.
    [100] Gururangan S, Swayamdipta S, Levy O, et al. Annotation artifacts in natural language inference data. In:Proc. of the NAACLHLT. 2018. 107-112.
    [101] Wang Z, Xu Y, Suo B, et al. A provenance storage method based on parallel database. In:Proc. of the Int'l Conf. on Information Science and Control Engineering. 2015. 63-66.
    [102] Liu C, Su T, Yu L, et al. Self-correction method for automatic data annotation. In:Proc. of the Asian Conf. on Pattern Recognition. 2017. 911-916.
    [103] Skala W, Wohlschlager T, Senn S, et al. MoFi:A software tool for annotating glycoprotein mass spectra by integrating hybrid data from the intact protein and glycopeptide level. Analytical Chemistry, 2018,90(9):5728-5736.
    [104] Zhu J, Zhang H, Guo J, et al. Data distributions automatic identification based on SOM and support vector machines. In:Proc. of the Int'l Conf. on Machine Learning and Cybernetics. 2002. 340-344.
    [105] Wang J. A review of China's statistical data quality research. In:Proc. of the 20162nd Fortune Today Forum. 2016. 393-394(in Chinese with English abstract).
    [106] Yang Y, He H, Wang D, et al. A framework to data delivery security for big data annotation delivery system. In:Proc. of the Mobile Adhoc and Sensor Systems. 2018. 532-536.
    [107] Verhulst SG. Where and when AI and CI meet:Exploring the intersection of artificial and collective intelligence towards the goal of innovating how we govern. AI and Society, 2018,33(2):293-297.
    [108] Ratner A, Varma P, Hancock B, et al. Weak supervision:A new programming paradigm for machine learning. In:Proc. of the Stanford AI Lab Blog. 2019.
    [109] Shetty R, Fritz M, Schiele B. Adversarial scene editing:Automatic object removal from weak supervision. arXiv:1806.01911, 2018.
    [110] Zhou Y, Nelakurthi AR, He J. Unlearn what you have learned:Adaptive crowd teaching with exponentially decayed memory learners. arXiv:1804.06481, 2018.
    附中文参考文献:
    [1] 轩中.人工智能行业中隐藏的"富士康"式劳动密集型产业.互联网周刊,2018,675(21):28-29.
    [20] 郭晓明,马良荔,苏凯,等.基于语义标注的数据资源库元数据质量自动评估方法研究.计算机应用与软件,2018,35(6):29-33,88.
    [24] 敬凯.社会化标注系统评价研究述评.江苏科技信息,2018,35(11):8-10.
    [26] 蔡莉,梁宇,朱扬勇,等.数据质量的历史沿革和发展趋势.计算机科学,2018,45(4):1-10.
    [30] 张博,郝杰,马刚,等.基于弱匹配概率典型相关性分析的图像自动标注.软件学报,2017,28(2):292-309. http://www.jos.org.cn/1000-98255047.htm[doi:10.13328/j.cnki.jos.005047]
    [57] 刘鹏,张燕.数据标注工程.北京:清华大学出版社,2019.
    [60] 李然,林政,林海伦,等.文本情绪分析综述.计算机研究与发展,2018,55(1):30-52.
    [61] 雷龙艳.中文微博细粒度情绪识别研究[硕士学位论文].衡阳:南华大学,2014.
    [62] 蔡莉,潘俊,魏宝乐,等.签到数据的热点区域时空模式与情感变化的可视化分析.小型微型计算机系统,2018,39(9):1889-1894.
    [63] 曹伟.众包域值标注算法研究[硕士学位论文].南京:南京财经大学,2017.
    [65] 徐太征,徐中宇.Fisher理论和多数投票法相结合的数据融合算法.科技信息,2009,27:445,488.
    [75] 于洪,陈云.基于Spark的3支聚类集成方法.郑州大学学报(理学版),2018,50(1):20-26.
    [79] 江跃华,丁磊,李娇娥,等.融合词汇特征的生成式摘要模型.河北科技大学学报,2019,40(2):152-158.
    [105] 王晶.中国统计数据质量研究综述.见:2016年第2届今日财富论坛.2016.393-394.
    引证文献
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

蔡莉,王淑婷,刘俊晖,朱扬勇.数据标注研究综述.软件学报,2020,31(2):302-320

复制
分享
文章指标
  • 点击次数:6606
  • 下载次数: 19874
  • HTML阅读次数: 8181
  • 引用次数: 0
历史
  • 收稿日期:2019-06-22
  • 最后修改日期:2019-09-17
  • 在线发布日期: 2019-12-06
  • 出版日期: 2020-02-06
文章二维码
您是第19763882位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62562563 传真:010-62562533 Email:jos@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号