利用标注者相关性的深度生成式众包学习
作者:
作者简介:

李绍园(1989-),女,博士,讲师,CCF专业会员,主要研究领域为机器学习,数据挖掘;
黄圣君(1987-),男,博士,教授,博士生导师,CCF专业会员,主要研究领域为机器学习,模式识别;
韦梦龙(1997-),男,硕士生,CCF学生会员,主要研究领域为机器学习,数据挖掘.

通讯作者:

李绍园,E-mail:lisy@nuaa.nju.edu.cn

基金项目:

国家自然科学基金(61906089);江苏省基础研究计划(BK20190408);中国博士后基金(2019TQ0152)


Deep Generative Crowdsourcing Learning with Worker Correlation Utilization
Author:
  • 摘要
  • | |
  • 访问统计
  • |
  • 参考文献 [38]
  • |
  • 相似文献 [20]
  • | | |
  • 文章评论
    摘要:

    传统监督学习需要训练样本的真实标记信息,而在很多情况下,真实标记并不容易收集.与之对比,众包学习从多个可能犯错的非专家收集标注,通过某种融合方式估计样本的真实标记.注意到现有深度众包学习工作对标注者相关性建模不足,而非深度众包学习方面的工作表明,标注者相关性建模利用有助于改善学习效果.提出一种深度生成式众包学习方法,以结合深度神经网络优势及利用标注者相关性.该模型由深度神经网络分类器先验和标注生成过程组成,其中,标注生成过程通过引入各类别内标注者能力的混合模型以建模标注者相关性.为自适应地匹配数据及模型复杂度,实现了完全贝叶斯推断.基于结构变分自编码器的自然梯度随机变分推断技术,将共轭参数变分消息传递与神经网络参数随机梯度下降结合到统一框架,实现端到端的高效优化.在22个真实众包数据集上的实验结果验证了该方法的有效性.

    Abstract:

    Traditional supervised learning requires the ground truth labels for the training data, which can be difficult to collect in many cases. In contrast, crowdsourcing learning collects noisy annotations from multiple non-expert workers and infers the latent true labels through some aggregation approach. This study notices that existing deep crowdsourcing work do not sufficiently model worker correlations, which however is shown to be helpful for learning by previous non-deep learning approaches. A deep generative crowdsourcing learning model is proposed to combine the strength of deep neural networks (DNN) and at the same time exploit the worker correlations. The model comprises a DNN classifier as a priori for the true labels, and one annotation generation process in which a mixture model of workers’ reliabilities within each class is introduced for inter-worker correlation. To automatically trade-off between the model complexity and data fitting, fully Bayesian inference is developed. Based on the natural-gradient stochastic variational inference techniques developed for structured variational autoencoder (SVAE), variational message passing is combined for conjugate parameters and stochastic gradient descent for DNN under a unified framework to conduct efficient end-to-end optimization. Experimental results on 22 real world crowdsourcing data sets demonstrate the effectiveness of the proposed approach.

    参考文献
    [1] Weld D, Lin C, Bragg J. Artificial intelligence and collective intelligence. In: Malone T, Bernstein M, eds. The Collective Intelligence Handbook. 2015.
    [2] Snow R, O’Connor B, Jurafsky D, et al. Cheap and fast—But is it good? Evaluating non-expert annotations for natural language tasks. In: Proc. of the Conf. on Empirical Methods in Natural Language Processing. 2008. 254-263.
    [3] Raykar V, Yu S, Zhao L, et al. Learning from crowds. Journal of Machine Learning Research, 2010, 11: 1297-1322.
    [4] Welinder P, Branson S, Belongie S, et al. The multidimensional wisdom of crowds. In: Lafferty J, Williams CI, Shawe-Taylor J, Zemel R, Culotta A, eds. Proc. of the Advances in Neural Information Processing Systems, Vol.23. 2010. 2024-2432.
    [5] Li Q, Li Y, Gao J, et al. A confidence-aware approach for truth discovery on long-tail data. Proc. of the VLDB Endowment, 2014, 8(4): 425-436.
    [6] Dawid AP, Skene AM. Maximum likeihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society, 1979, 28(1): 20-28.
    [7] Liu Q, Peng J, Ihler A. Variational inference for crowdsourcing. In: Bartlett P, Pereira F, Burges C, Bottou L, Weinberger K, eds. Proc. of the Advances in Neural Information Processing Systems, Vol.25. 2012. 692-700.
    [8] Whitehill J, Ruvolo P, Wu T, et al. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In: Bengio Y, Schuurmans D, Lafferty J, Williams C, Culotta A, eds. Proc. of the Advances in Neural Information Processing Systems, Vol.22. 2009. 2035-2043.
    [9] Zhou D, Basu S, Mao Y, et al. Learning from the wisdom of crowds by minimax entropy. In: Bartlett P, Pereira F, Burges C, Bottou L, Weinberger K, eds. Proc. of the Advances in Neural Information Processing Systems, Vol.25. 2012. 2195-2203.
    [10] Kim HC, Ghahramani Z. Bayesian classifier combination. In: Proc. of the 15th Int’l Conf. on Artificial Intelligence and Statistics. 2012. 619-627.
    [11] Imamura H, Sato I, Sugiyama M. Analysis of minimax error rate for crowdsourcing and its application to worker clustering model. In: Proc. of the 35th Int’l Conf. on Machine Learning. 2018. 2152-2161.
    [12] Li Y, Rubinstein BIP, Cohn T. Exploiting worker correlation for label aggregation in crowdsourcing. In: Proc. of the 36th Int’l Conf. on Machine Learning. 2019. 3886-3895.
    [13] Rodrigues F, Pereira F, Ribeiro B. Gaussian process classification and active learning with multiple annotators. In: Proc. of the 31th Int’l Conf. on Machine Learning. 2014. 433-441.
    [14] LeCun Y, Bengio Y, Hinton GE. Deep learning. Nature, 2015, 521(7553): 436-444.
    [15] Albarqouni S, Baur C, Achilles F, et al. Aggnet: Deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans. on Medical Imaging, 2016, 35(5): 1313-1321.
    [16] Filipe Rodrigues FCP. Deep learning from crowds. In: Proc. of the 32nd AAAI Conf. on Artificial Intelligence. 2018. 1611-1618.
    [17] Tanno R, Saeedi A, Sankaranarayanan S, et al. Learning from noisy labels by regularized estimation of annotator confusion. In: Proc. of the 2019 IEEE Conf. on Computer Vision and Pattern Recognition. 2019. 11244-11253.
    [18] Kingma DP, Welling M. Auto-Encoding variational Bayes. In: Proc. of the 2nd Int’l Conf. on Learning Representations. 2014.
    [19] Kingma DP, Mohamed S, Rezende DJ, et al. Semi-Supervised learning with deep generative models. In: Ghahramani Z, Welling M, Cortes C, Lawrence ND, Weinberger KQ, eds. Proc. of the Advances in Neural Information Processing Systems, Vol.27. 2014. 3581-3589.
    [20] Johnson MJ, Duvenaud D, Wiltschko AB, et al. Composing graphical models with neural networks for structured representations and fast inference. In: Lee DD, Sugiyama M, von Luxburg U, Guyon I, Garnett R, eds. Proc. of the Advances in Neural Information Processing Systems, Vol.29. 2016. 2946-2954.
    [21] Yin L, Han J, Zhang W, et al. Aggregating crowd wisdoms with label-aware autoencoders. In: Proc. of the 26th Int’l Joint Conf. on Artificial Intelligence. 2017. 1325-1331.
    [22] Atarashi K, Oyama S, Kurihara M. Semi-supervised learning from crowds using deep generative models. In: Proc. of the 32nd AAAI Conf. on Artificial Intelligence. 2018. 1555-1562.
    [23] Shi W, Sheng VS, Li X, et al. Semi-supervised multi-label learning from crowds via deep sequential generative model. In: Proc. of the 26th ACM SIGKDD Int’l Conf. on Knowledge Discovery and Data Mining. 2020. 1141-1149.
    [24] Luo Y, Tian T, Shi J, et al. Semi-crowdsourced clustering with deep generative models. In: Bengio S, Wallach HM, Larochelle H, Grauman K, Cesa-Bianchi N, Garnett R, eds. Proc. of the Advances in Neural Information Processing Systems, Vol.31. 2018. 3216-3226.
    [25] Li SY, Huang SJ, Chen S. Crowdsourcing aggregation with deep Bayesian learning. Science China Information Sciences, 2021, 64(3).
    [26] Simpson E, Roberts S, Psorakis I, et al. Dynamic Bayesian combination of multiple imperfect classiers. In: Proc. of the Decision Making and Imperfection. 2013. 1-35.
    [27] Venanzi M, Guiver J, Kazai G, et al. Community-based Bayesian aggregation models for crowdsourcing. In: Proc. of the 23rd Int’l Conf. on World Wide Web. 2014. 155-164.
    [28] Homan MD, Blei DM, Wang C, et al. Stochastic variational inference. Journal of Machine Learning Research, 2013,14(1): 1303-1347.
    [29] Li SY, Jiang Y. Multi-Label crowdsourcing learning. Ruan Jian Xue Bao/Journal of Software, 2020, 31(5): 1497-1510 (in Chinese with English abstract). http://www.jos.org.cn/1000-9825/5673.htm [doi: 10.13328/j.cnki.jos.005673]
    [30] Chen CJ, Jiang L, Lei N, et al. An interactive feature selection method based on learning-from-crowds. Sci Sin Inform, 2020, 50: 794-812 (in Chinese with English abstract). [doi: 10.1360/SSI-2019-0208]
    [31] Feng Y, Wang Y, Fang CR, et al. An approach for developing a highly trustworthy crowd-sourced workforce. Sci Sin Inform, 2019, 49: 1412-1427 (in Chinese with English abstract). [doi: 10.1360/N112018-00303]
    [32] Li SY, Jiang Y, Chawla NV, et al. Multi-Label learning from crowds. IEEE Trans. on Knowledge and Data Engineering, 2019, 31(7): 1369-1382.
    [33] Kingma DP, Ba J. Adam: A method for stochastic optimization. In: Proc. of the 3rd Int’l Conf. on Learning Representations. CA, 2015.
    [34] Demsar J. Statistical comparisons of classifiers over multiple datasets. Journal of Machine Learning Research, 2006, 7: 1-30.
    附中文参考文献:
    [29] 李绍园, 姜远. 多标记众包学习. 软件学报, 2020, 31(5): 1497-1510. http://www.jos.org.cn/1000-9825/5673.htm [doi: 10.13328/ j.cnki.jos.005673]
    [30] 陈长建, 姜流, 雷娜, 等. 基于众包学习的交互式特征选择方法. 中国科学: 信息科学, 2020, 50: 794-812. [doi: 10.1360/SSI- 2019-0208]
    [31] 冯洋, 王祎, 房春荣, 等. 高可信众包群体构建方法. 中国科学: 信息科学, 2019, 49: 1412-1427. [doi: 10.1360/N112018- 00303]
    引证文献
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

李绍园,韦梦龙,黄圣君.利用标注者相关性的深度生成式众包学习.软件学报,2022,33(4):1274-1286

复制
分享
文章指标
  • 点击次数:1132
  • 下载次数: 4691
  • HTML阅读次数: 2525
  • 引用次数: 0
历史
  • 收稿日期:2021-05-31
  • 最后修改日期:2021-07-16
  • 在线发布日期: 2021-10-26
  • 出版日期: 2022-04-06
文章二维码
您是第19705282位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62562563 传真:010-62562533 Email:jos@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号