微信服务号

微信订阅号

2025年4月5日 22:10 星期六
首页 > 过刊浏览>2018年第29卷第4期 >1131-1142. DOI:10.13328/j.cnki.jos.005263
PDF HTML阅读 XML下载 导出引用 引用提醒
基于权值不确定性的玻尔兹曼机算法
作者:
作者简介:

丁世飞(1963-),男,山东青岛人,博士,教授,博士生导师,CCF杰出会员,主要研究领域为智能信息处理,人工智能,模式识别,机器学习,数据挖掘,粗糙集,软计算,大数据分析,云计算;张健(1990-),男,学士,主要研究领域为机器学习,模式识别;史忠植(1941-),男,博士,教授,博士生导师,CCF会士,主要研究领域为智能科学,人工智能,机器学习.

通讯作者:

丁世飞,E-mail:dingsf@cumt.edu.cn

基金项目:

国家自然科学基金（61672522，61379101）；国家重点基础研究发展计划（973）（2013CB329502）


Algorithms of Boltzmann Machines Based on Weight Uncertainty
Author:
Fund Project:

National Natural Science Foundation of China (61672522, 61379101); National Basic Research Program of China (973) (2013CB329502)

  • 摘要
    • | |
  • 访问统计
    • |
  • 参考文献 [32]
    • |
  • 相似文献
    • |
  • 引证文献
    • | |
  • 文章评论
    摘要:

    受限制的玻尔兹曼机（RBM）是一种无向图模型.基于RBM的深度学习模型包括深度置信网（DBN）和深度玻尔兹曼机（DBM）等.在神经网络和RBM的训练过程中，过拟合问题是一个比较常见的问题.针对神经网络的训练，权值随机变量（weight random variables）、Dropout方法和早期停止方法已被用于缓解过拟合问题.首先，改变RBM模型中的训练参数，使用随机变量代替传统的实值变量，构建了基于随机权值的受限的波尔兹曼机（weight uncertainty RBM，简称WRBM），接下来，在WRBM基础上构建了相应的深度模型：Weight uncertainty Deep Belief Network（WDBN）和Weight uncertainty Deep Boltzmann Machine（WDBM），并且通过实验验证了WDBN和WDBM的有效性.最后，为了更好地建模输入图像，引入基于条件高斯分布的RBM模型，构建了基于spike-and-slab RBM（ssRBM）的深度模型，并通过实验验证了模型的有效性.

    Abstract:

    Based on the restricted Boltzmann machine (RBM), which is a probabilistic graphical model, deep learning models contain deep belief net (DBN) and deep Boltzmann machine (DBM). The overfitting problems commonly exist in neural networks and RBM models. In order to alleviate the overfitting problem, this paper introduces weight random variables to the conventional RBM model and, then builds weight uncertainty deep models based on maximum likelihood estimation. In the experimental section, the paper verifies the effectiveness of the weight uncertainty RBM. In order to improve the image recognition ability, the paper introduces the spike-and-slab RBM (ssRBM) to weight uncertainty RBM and then builds the deep models. The experiments show that the deep models based on weight random variables are effective.

    参考文献
    [1] Erhan D, Bengio Y, Courville A, Manzagol PA, Vincent P, Bengio S. Why does unsupervised pre-training help deep learning. Journal of Machine Learning Research, 2010,11(3):625-660.
    [2] Hinton GE. Training products of experts by minimizing contrastive divergence. Neural Computation, 2002,14(8):1771-1800.[doi:10.1162/089976602760128018]
    [3] Roux N, Bengio Y. Representational power of restricted Boltzmann machines and deep belief networks. Neural Computation, 2008, 20(6):1631-1649.[doi:10.1162/neco.2008.04-07-510]
    [4] Liu JW, Liu Y, Luo XL. Research and development on Boltzmann machine. Journal of Computer Research and Development, 2014,51(1):1-16(in Chinese with English abstract).[doi:10.7544/issn1000-1239.2014.20121044]
    [5] Hinton GE, Osindero S, Th Y. A fast learning algorithm for deep belief nets. Neural Computation, 2006,18(7):1527-1554.[doi:10.1162/neco.2006.18.7.1527]
    [6] Hinton GE, Salakhutdinov R. Reducing the dimensionality of data with neural networks. Science, 2006,313(5786):504-507.[doi:10.1126/science.1127647]
    [7] Lee H, Pham PT, Yan L, Ng A. Unsupervised feature learning for audio classification using convolutional deep belief networks. In:Bengio Y, Schuurmans D, Lafferty JD, Williams CKI, Culotta A, eds. Proc. of the Advances in Neural Information Processing Systems. 2009. 1096-1104. https://papers.nips.cc/book/advances-in-neural-information-processing-systems-22-2009
    [8] Norouzi M, Ranjbar M, Mori G. Stacks of convolutional restricted Boltzmann machines for shift-invariant feature learning. In:Proc. of the Computer Vision and Pattern Recognition. 2009. 2735-2742.[doi:10.1109/CVPR.2009.5206577]
    [9] Salakhutdinov R, Larochelle H. Efficient learning of deep Boltzmann machines. Journal of Machine Learning Research, 2010,9(8):693-700.
    [10] Salakhutdinov R, Hinton GE. An efficient learning procedure for deep Boltzmann machines. Neural Computation, 2012,24(8):1967-2006.[doi:10.1162/NECO_a_00311]
    [11] Boulanger-Lewandowski N, Bengio Y, Vincent P. Modeling temporal dependencies in high-dimensional sequences:Application to polyphonic music generation and transcription. Chemistry a European Journal, 2012,18(13):3981-3991.
    [12] Hu Z, Fu K, Zhang CS. Audio classical composer identification by deep neural network. Journal of Computer Research and Development, 2014,51(9):1945-1954(in Chinese with English abstract).[doi:10.7544/issn1000-1239.2014.20140189]
    [13] Zhang J, Ding SF, Zhang N, Shi ZZ. Incremental extreme learning machine based on deep feature embedded. Int'l Journal of Machine Learning and Cybernetics, 2016,7(1):111-120.[doi:10.1007/s13042-015-0419-5]
    [14] Zhang N, Ding SF, Shi ZZ. Denoising Laplacian multi-layer extreme learning machine. Neurocomputing, 2016,171(C):1066-1074.[doi:10.1016/j.neucom.2015.07.058]
    [15] Ding SF, Zhang N, Xu XZ, Guo LL, Zhang J. Deep extreme learning machine and its application in EEG classification. In:Proc. of the Mathematical Problems in Engineering. 2015. 1-11.[doi:10.1155/2015/129021]
    [16] Srivastava N, Hinton GE, Krizhevsky A. Dropout:A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 2014,15(1):1929-1958.
    [17] Blundell C, Cornebise J, Kavukcuoglu K. Weight uncertainty in neural networks. In:Bach F, Blei D, eds. Proc. of the 32nd Int'l Conf. on Machine Learning. Lille, 2015. http://proceedings.mlr.press/v37/
    [18] Hinton GE. A practical guide to training restricted Boltzmann machines. Momentum, 2010,9(1):926.[doi:10.1007/978-3-642-35289-8_32]
    [19] Krizhevsky A, Hinton GE. Learning multiple layers of features from tiny images. Technical Report, University of Toronto, 2009.
    [20] Salakhutdinov RR. Learning in Markov random fields using tempered transitions. In:Bengio Y, Schuurmans D, Lafferty JD, et al., eds. Proc. of the Advances in Neural Information Processing Systems. Curran Associates Inc., 2009. 1598-1606.
    [21] Desjardins G, Courville AC, Bengio Y, Vincent P, Delalleau O. Tempered Markov chain Monte Carlo for training of restricted Boltzmann machines. In:Lawrence N, Whye TY, Titterington M, eds. Proc. of the Int'l Conf. on Artificial Intelligence and Statistics, Vol.9. 2010. 145-152.
    [22] Peterson C. A mean field theory learning algorithm for neural network. Complex Systems, 1987,1(3):995-1019.
    [23] Tieleman T. Training restricted Boltzmann machines using approximations to the likelihood gradient. In:Proc. of the 25th Int'l Conf. on Machine Learning. ACM Press, 2008. 1064-1071.[doi:10.1145/1390156.1390290]
    [24] Tieleman T, Hinton GE. Using fast weights to improve persistent contrastive divergence. In:Proc. of the 26th Int'l Conf. on Machine Learning. ACM Press, 2009. 1033-1040.[doi:10.1145/1553374.1553506]
    [25] Bengio Y. Learning deep architectures for AI. Foundations & Trends in Machine Learning, 2009,2(1):1-127.[doi:10.1561/2200000006]
    [26] Lee T, Yoon S. Boosted categorical restricted Boltzmann machine for computational prediction of splice junctions. In:Bach F, Blei D, eds. Proc. of the Int'l Conf. on Machine Learning. 2015. 2483-2492.
    [27] Ranzato M, Krizhevsky A, Hinton GE. Factored 3-way restricted Boltzmann machines for modeling natural images. Journal of Machine Learning Research, 2010,9:621-628.
    [28] Courville AC, Bergstra J, Bengio Y. A spike and slab restricted Boltzmann machine. Journal of Machine Learning Research, 2011, 15(15):233-241.
    [29] Courville AC, Desjardins G, Bergstra J, Bengio Y. The spike-and-slab RBM and extensions to discrete and sparse data distributions. IEEE Trans. on Software Engineering, 2014,36(9):1874-1887.[doi:10.1109/TPAMI.2013.238]
    附中文参考文献:
    [4] 刘建伟,刘媛,罗雄麟.玻尔兹曼机研究进展.计算机研究与发展,2014,51(1):1-16.[doi:10.7544/issn1000-1239.2014.20121044]
    [12] 胡振,傅昆,张长水.基于深度学习的作曲家分类问题.计算机研究与发展,2014,51(9):1945-1954.[doi:10.7544/issn1000-1239. 2014.20140189]
    相似文献
    引证文献
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

丁世飞,张健,史忠植.基于权值不确定性的玻尔兹曼机算法.软件学报,2018,29(4):1131-1142

复制
分享
文章指标
  • 点击次数:3520
  • 下载次数: 5001
  • HTML阅读次数: 1688
  • 引用次数: 0
历史
  • 收稿日期:2016-09-06
  • 最后修改日期:2016-10-19
  • 在线发布日期: 2017-04-11
文章二维码
友情链接:中国科学院软件研究所中国计算机学会中国科学院国家自然科学基金委员会CCF数字图书馆Int'l J of Softw Info计算机系统应用
您是第19791893位访问者
版权所有：中国科学院软件研究所 京ICP备05046678号-3
地址：北京市海淀区中关村南四街4号,邮政编码：100190
电话：010-62562563 传真：010-62562533 Email：jos@iscas.ac.cn
技术支持：北京勤云科技发展有限公司

京公网安备 11040202500063号