微信服务号

微信订阅号

2025年6月1日 18:16 星期日
首页 > 过刊浏览>2021年第32卷第2期 >406-423. DOI:10.13328/j.cnki.jos.006147
PDF HTML阅读 XML下载 导出引用 引用提醒
机器学习安全攻击与防御机制研究进展和未来挑战
作者:
作者简介:

李欣姣(1992-),女,博士生,主要研究领域为大数据安全和隐私.
吴国伟(1973-),男,博士,教授,博士生导师,CCF专业会员,主要研究领域为自主智能系统,智能边缘计算.
姚琳(1976-),女,博士,教授,博士生导师,CCF专业会员,主要研究领域为大数据安全和隐私.
张伟哲(1976-),男,博士,教授,博士生导师,CCF杰出会员,主要研究领域为网络空间安全,网络安全,系统安全,内容安全,云计算,高性能计算.
张宾(1976-),男,博士,高级工程师,CCF专业会员,主要研究领域为网络测量,拓扑发现,异常检测.

通讯作者:

吴国伟,E-mail:wgwdut@dlut.edu.cn

基金项目:

国家自然科学基金（61872053）；中央高校基本科研业务费专项资金（DUT19GJ204）；广东省重点领域研发计划（2019B010136001）；广东省重点科技计划（LZC0023）


Progress and Future Challenges of Security Attacks and Defense Mechanisms in Machine Learning
Author:
Fund Project:

National Natural Science Foundation of China (61872053); Fundamental Research Funds for the Central Universities (DUT19GJ204); Key-Area Research and Development Program of Guangdong Province (2019B010136001); Key Science and Technology Program of Guangdong Province (LZC0023)

  • 摘要
    • | |
  • 访问统计
    • |
  • 参考文献 [100]
    • |
  • 相似文献
    • | | |
  • 文章评论
    摘要:

    机器学习的应用遍及人工智能的各个领域，但因存储和传输安全问题以及机器学习算法本身的缺陷，机器学习面临多种面向安全和隐私的攻击.基于攻击发生的位置和时序对机器学习中的安全和隐私攻击进行分类，分析和总结了数据投毒攻击、对抗样本攻击、数据窃取攻击和询问攻击等产生的原因和攻击方法，并介绍和分析了现有的安全防御机制.最后，展望了安全机器学习未来的研究挑战和方向.

    Abstract:

    Machine learning applications span all areas of artificial intelligence, but due to storage and transmission security issues and the flaws of machine learning algorithms themselves, machine learning faces a variety of security- and privacy-oriented attacks. This survey classifies the security and privacy attacks based on the location and timing of attacks in machine learning, and analyzes the causes and attack methods of data poisoning attacks, adversary attacks, data stealing attacks, and querying attacks. Furthermore, the existing security defense mechanisms are summarized. Finally, a perspective of future work and challenges in this research area are discussed.

    参考文献
    [1] Silver D, Huang A, Maddison CJ, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 2016, 529(7587):484-489.[doi:10.1038/nature16961]
    [2] Dalvi N, Domingos P, Sanghai S, Verma D, et al. Adversarial classification. In:Proc. of the 10th ACM SIGKDD Int'l Conf. on Knowledge Discovery and Data Mining. ACM, 2004. 99-108.[doi:10.1145/1014052.1014066]
    [3] Lowd D, Meek C. Adversarial learning. In:Proc. of the 11th ACM Sigkdd Int'l Conf. on Knowledge Discovery in Data Mining. 2005.[doi:10.1145/1081870.1081950]
    [4] Kearns MJ, Li M. Learning in the presence of malicious errors. SIAM Journal on Computing, 1993,22(4):807-837.[doi:10.1137/0222052]
    [5] Szegedy C, Zaremba W, Sutskever I, et al. Intriguing properties of neural networks. In:Proc. of the Int'l Conf. on Learning Representations. 2014.
    [6] Carlini N, Wagner D. Towards evaluating the robustness of neural networks. In:Proc. of the IEEE Symp. on Security and Privacy (SP). 2017. 39-57.[doi:10.1109/sp.2017.49]
    [7] Papernot N, Mcdaniel P, Sinha A, et al. SoK:Security and privacy in machine learning. In:Proc. of the IEEE European Symp. on Security and Privacy. 2018. 399-414.[doi:10.1109/EuroSP.2018.00035]
    [8] Papernot N, Mcdaniel PD, Jha S, et al. The limitations of deep learning in adversarial settings. In:Proc. of the IEEE European Symp. on Security and Privacy. 2016. 372-387.[doi:10.1109/EuroSP.2016.36]
    [9] Song L, Ma CG, Duan GH. Machine learning security and privacy:A survey. Chinese Journal of Network and Information Security, 2018,4(8):1-11(in Chinese with English abstract).[doi:10.11959/j.issn.2096-109x.2018067]
    [10] Kurakin A, Goodfellow IJ, Bengio S, et al. Adversarial machine learning at scale. In:Proc. of the Int'l Conf. on Learning Representations. 2017.
    [11] Li P, Zhao WT, Liu Q, et al. Security issues and their countermeasuring techniques of machine learning:A survey. Journal of Frontiers of Computer Science and Technology, 2018,12(2):171-184(in Chinese with English abstract).[doi:10.3778/j.issn.1673-9418.1708038]
    [12] Biggio B, Corona I, Maiorca D, et al. Evasion attacks against machine learning at test time. In:Proc. of the European Conf. on Machine Learning. 2013. 387-402.[doi:10.1007/978-3-642-40994-3_25]
    [13] Powers DM. Evaluation:From precision, recall and f-measure to roc, informedness, markedness and correlation. Journal of Machine Learning Technologies, 2011,2(1):37-63.[doi:10.9735/2229-3981]
    [14] Biggio B, Nelson B, Laskov P, et al. Poisoning attacks against support vector machines. In:Proc. of the Int'l Conf. on Machine Learning. 2012. 1467-1474.
    [15] Mahloujifar S, Mahmoody M, Mohammed A, et al. Multi-party poisoning through generalized p-tampering. arXiv:1809.0347, 2018.
    [16] Rubinstein BI, Nelson B, Huang L, et al. ANTIDOTE:Understanding and defending against poisoning of anomaly detectors. In:Proc. of the Internet Measurement Conf. 2009. 1-14.[doi:10.1145/1644893.1644895]
    [17] Jacob S, Pang WK, Percy L. Certified defenses for data poisoning attacks. In:Proc. of the 31st Int'l Conf. on Neural Information Processing Systems. 2017. 3520-3532.
    [18] Sharif M, Bhagavatula S, Bauer L, et al. Accessorize to a Crime:Real and stealthy attacks on state-of-the-art face recognition. In:Proc. of the Computer and Communications Security. 2016. 1528-1540.[doi:10.1145/2976749.2978392]
    [19] Globerson A, Roweis ST. Nightmare at test time:Robust learning by feature deletion. In:Proc. of the Int'l Conf. on Machine Learning. 2006. 353-360.[doi:10.1145/1143844.1143889]
    [20] Valiant LG. A theory of the learnable. Symp. on the Theory of Computing, 1984,27(11):1134-1142.[doi:10.1145/1968.1972]
    [21] Biggio B, Nelson B, Laskov P, et al. Support vector machines under adversarial label noise. In:Proc. of the Asian Conf. on Machine Learning. 2011. 97-112.
    [22] Mozaffarikermani M, Surkolay S, Raghunathan A, et al. Systematic Poisoning attacks on and defenses for machine learning in healthcare. IEEE Journal of Biomedical and Health Informatics, 2014,19(6):1893-1905.[doi:10.1109/JBHI.2014.2344095]
    [23] Mei S, Zhu X. Using machine teaching to identify optimal training-set attacks on machine learners. In:Proc. of the National Conf. on Artificial Intelligence. 2015. 2871-2877.
    [24] Kloft M, Laskov P. Online anomaly detection under adversarial impact. In:Proc. of the 13th Int'l Conf. on Artificial Intelligence and Statistics (AISTATS). 2010. 405-412.
    [25] Kloft M, Laskov P. Security analysis of online centroid anomaly detection. Journal of Machine Learning Research, 2012,13(1):3681-3724.[doi:10.1016/j.dss.2012.08.019]
    [26] Biggio B, Didaci L, Fumera G, et al. Poisoning attacks to compromise face templates. In:Proc. of the Int'l Conf. on Biometrics. 2013. 1-7.[doi:10.1109/ICB.2013.6613006]
    [27] Perdisci R, Dagon D, Lee W, et al. Misleading worm signature generators using deliberate noise injection. In:Proc. of the IEEE Symp. on Security and Privacy. 2006. 17-31.[doi:10.1109/SP.2006.26]
    [28] Kurakin A, Goodfellow IJ, Bengio S, et al. Adversarial examples in the physical world. In:Proc. of the Int'l Conf. on Learning Representations. 2017.
    [29] Papernot N, Mcdaniel P, Goodfellow I, et al. Practical black-box attacks against machine learning. In:Proc. of the Computer and Communications Security. 2017. 506-519.[doi:10.1145/3052973.3053009]
    [30] Athalye A, Engstrom L, Ilyas A, et al. Synthesizing robust adversarial examples. In:Proc. of the Int'l Conf. on Machine Learning. 2018. 284-293.
    [31] Moosavidezfooli S, Fawzi A, Fawzi O, et al. Universal adversarial perturbations. In:Proc. of the Computer Vision and Pattern Recognition. 2017. 86-94.[doi:10.1109/CVPR.2017.17]
    [32] Grosse K, Papernot N, Manoharan P, et al. Adversarial perturbations against deep neural networks for malware classification. arXiv:1606.04435, 2016.
    [33] Moosavidezfooli S, Fawzi A, Frossard P, et al. DeepFool:A simple and accurate method to fool deep neural networks. In:Proc. of the Computer Vision and Pattern Recognition. 2016. 2574-2582.[doi:10.1109/CVPR.2016.282]
    [34] Huang R, Xu B, Schuurmans D, et al. Learning with a strong adversary. arXiv:1511.03034, 2015.
    [35] Nguyen A, Yosinski J, Clune J, et al. Deep neural networks are easily fooled:High confidence predictions for unrecognizable images. In:Proc. of the Computer Vision and Pattern Recognition. 2015. 427-436.[doi:10.1109/CVPR.2015.7298640]
    [36] Carlini N, Mishra P, Vaidya T, et al. Hidden voice commands. In:Proc. of the Usenix Security Symp. 2016. 513-530.
    [37] Smith DF, Wiliem A, Lovell BC, et al. Face recognition on consumer devices:Reflections on replay attacks. IEEE Trans. on Information Forensics and Security, 2015,10(4):736-745.[doi:10.1109/TIFS.2015.2398819]
    [38] Sharif M, Bhagavatula S, Bauer L, et al. Accessorize to a crime:Real and stealthy attacks on state-of-the-art face recognition. In:Proc. of the Computer and Communications Security. 2016. 1528-1540.[doi:10.1145/2976749.2978392]
    [39] Wittel GL, Wu SF. On attacking statistical spam filters. In:Proc. of the Conf. on Email and Anti-Spam. 2004. 1-7.
    [40] Lowd D, Meek C. Good word attacks on statistical spam filters. In:Proc. of the Conf. on Email and Anti-Spam. 2005. 1-8.
    [41] Alfeld S, Zhu X, Barford P, et al. Data poisoning attacks against autoregressive models. In:Proc. of the National Conf. on Artificial Intelligence. 2016. 1452-1458.
    [42] Xu W, Qi Y, Evans D, et al. Automatically evading classifiers:A case study on PDF malware classifiers. In:Proc. of the Network and Distributed System Security Symp. 2016.[doi:10.14722/ndss.2016.23115]
    [43] Smutz C, Stavrou A. Malicious PDF detection using metadata and structural features. In:Proc. of the Annual Computer Security Applications Conf. 2012. 239-248.[doi:10.1145/2420950.2420987]
    [44] Srndic N, Laskov P. Detection of malicious PDF files based on hierarchical document structure. In:Proc. of the Network and Distributed System Security Symp. 2013.[doi:10.1145/2420950.2420987]
    [45] Rndic N, Laskov P. Practical evasion of a learning-based classifier:A case study. In:Proc. of the IEEE Symp. on Security and Privacy. 2014. 197-211.[doi:10.1109/SP.2014.20]
    [46] Smutz C, Stavrou A. Malicious PDF detection using metadata and structural features. In:Proc. of the Annual Computer Security Applications Conf. 2012. 239-248.[doi:10.1145/2420950.2420987]
    [47] Papernot N, Mcdaniel PD, Goodfellow IJ, et al. Practical black-box attacks against deep learning systems using adversarial examples. arXiv:1602.02697v2, 2016.
    [48] Papernot N, Mcdaniel PD, Goodfellow IJ, et al. Transferability in machine learning:From phenomena to black-box attacks using adversarial samples. arXiv:1605.07277, 2016.
    [49] Papernot N, Mcdaniel PD, Sinha A, et al. Towards the science of security and privacy in machine learning. arXiv:1611.03814, 2016.
    [50] Alrubaie M, Chang JM. Privacy-preserving machine learning:Threats and solutions. IEEE Symp. on Security and Privacy, 2019, 17(2):49-58.[doi:10.1109/MSEC.2018.2888775]
    [51] Feng J, Jain AK. Fingerprint reconstruction:From minutiae to phase. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2011,33(2):209-223.[doi:10.1109/TPAMI.2010.77]
    [52] Alrubaie M, Chang JM. Reconstruction attacks against mobile-based continuous authentication systems in the cloud. IEEE Trans. on Information Forensics and Security, 2016,11(12):2648-2663.[doi:10.1109/TIFS.2016.2594132]
    [53] Shokri R, Stronati M, Song C, et al. Membership inference attacks against machine learning models. In:Proc. of the IEEE Symp. on Security and Privacy. 2017. 3-18.[doi:10.1109/SP.2017.41]
    [54] Fredrikson M, Lantz E, Jha S, et al. Privacy in pharmacogenetics:An end-to-end case study of personalized warfarin dosing. In:Proc. of the Usenix Security Symp. 2014. 17-32.
    [55] Fredrikson M, Jha S, Ristenpart T, et al. Model inversion attacks that exploit confidence information and basic countermeasures. In:Proc. of the Computer and Communications Security. 2015. 1322-1333.[doi:10.1145/2810103.2813677]
    [56] Vorobeychik Y, Li B. Optimal randomized classification in adversarial settings. In:Proc. of the Int'l Conf. on Autonomous Agents and Multi-agent Systems. 2014. 485-492.
    [57] Ateniese G, Mancini LV, Spognardi A, et al. Hacking smart machines with smarter ones:How to extract meaningful data from machine learning classifiers. Int'l Journal of Security and Networks, 2015,10(3):137-150.[doi:10.1504/ijsn.2015.071829]
    [58] Tramer F, Zhang F, Juels A, et al. Stealing machine learning models via prediction APIs. In:Proc. of the Usenix Security Symp. 2016. 601-618.
    [59] Qiao M, Valiant G. Learning discrete distributions from untrusted batches. In:Proc. of the Conf. on Innovations in Theoretical Computer Science. 2018.[doi:10.4230/LIPIcs.ITCS.2018.47]
    [60] Steinhardt J, Koh PW, Liang P, et al. Certified defenses for data poisoning attacks. In:Proc. of the Neural Information Processing Systems. 2017. 3517-3529.
    [61] Rubinstein BI, Nelson B, Huang L, et al. ANTIDOTE:Understanding and defending against poisoning of anomaly detectors. In:Proc. of the Internet Measurement Conf. 2009. 1-14.[doi:10.1145/1644893.1644895]
    [62] Fung C, Yoon CJ, Beschastnikh I, et al. Mitigating sybils in federated learning poisoning. arXiv:1808.04866, 2018.
    [63] Shen SQ, Tople S, Saxena P. Auror:Defending against poisoning attacks in collaborative deep learning systems. In:Proc. of the Conf. on Computer Security Applications. 2016. 508-519.[doi:10.1145/2991079.2991125]
    [64] Gu S, Rigazio L. Towards deep neural network architectures robust to adversarial examples. In:Proc. of the Int'l Conf. on Learning Representations. 2014.
    [65] Srivastava N, Hinton GE, Krizhevsky A, et al. Dropout:A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 2014,15(1):1929-1958.
    [66] Pinto L, Davidson J, Gupta A, et al. Supervision via competition:Robot adversaries for learning tasks. In:Proc. of the Int'l Conf. on Robotics and Automation. 2017. 1601-1608.[doi:10.1109/ICRA.2017.7989190]
    [67] Goodfellow I, Shlens J, Szegedy C, et al. Explaining and harnessing adversarial examples. In:Proc. of the Int'l Conf. on Learning Representations. 2015.
    [68] Huang R, Xu B, Schuurmans D, et al. Learning with a strong adversary. arXiv:1511.03034, 2015.
    [69] Tramer F, Kurakin A, Papernot N, et al. Ensemble adversarial training:Attacks and defenses. In:Proc. of the Int'l Conf. on Learning Representations. 2018.
    [70] Carlini N, Wagner DA. Towards evaluating the robustness of neural networks. In:Proc. of the IEEE Symp. on Security and Privacy. 2017. 39-57.[doi:10.1109/SP.2017.49]
    [71] Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network. arXiv:1503.02531, 2015.
    [72] Papernot N, Mcdaniel P, Wu X, et al. Distillation as a defense to adversarial perturbations against deep neural networks. In:Proc. of the IEEE Symp. on Security and Privacy. 2016. 582-597.[doi:10.1109/SP.2016.41]
    [73] Athalye A, Carlini N, Wagner D. Obfuscated gradients give a false sense of security:Circumventing defenses to adversarial examples. In:Proc. of the Int'l Conf. on Machine Learning. 2018.
    [74] Papernot N, Mcdaniel P. Extending defensive distillation. arXiv:1705.05264, 2017.
    [75] Carlini N, Wagner DA. Defensive distillation is not robust to adversarial examples. arXiv:1607.04311, 2016.
    [76] Bost R, Popa RA, Tu S, et al. Machine learning classification over encrypted data. In:Proc. of the Network and Distributed System Security Symp. 2015.[doi:10.14722/ndss.2015.23241]
    [77] Nikolaenko V, Weinsberg U, Ioannidis S, et al. Privacy-preserving ridge regression on hundreds of millions of records. In:Proc. of the IEEE Symp. on Security and Privacy. 2013. 334-348.[doi:10.1109/SP.2013.30]
    [78] Bogdanov D, Kamm L, Laur S, et al. Implementation and evaluation of an algorithm for cryptographically private principal component analysis on genomic data. IEEE/ACM Trans. on Computational Biology and Bioinformatics, 2018,15(5):1427-1432.[doi:10.1109/TCBB.2018.2858818]
    [79] Bonawitz KA, Ivanov V, Kreuter B, et al. Practical secure aggregation for privacy-preserving machine learning. In:Proc. of the Computer and Communications Security. 2017. 1175-1191.[doi:10.1145/3133956.3133982]
    [80] Erkin Z, Veugen T, Toft T, et al. Generating private recommendations efficiently using homomorphic encryption and data packing. IEEE Trans. on Information Forensics and Security, 2012,7(3):1053-1066.[doi:10.1109/TIFS.2012.2190726]
    [81] Bost R, Popa RA, Tu S, et al. Machine learning classification over encrypted data. In:Proc. of the Network and Distributed System Security Symp. 2015.[doi:10.14722/ndss.2015.23241]
    [82] Ohrimenko O, Schuster F, Fournet C, et al. Oblivious multi-party machine learning on trusted processors. In:Proc. of the Usenix Security Symp. 2016. 619-636.
    [83] Dwork C. Differential privacy. In:Proc. of the Int'l Colloquium on Automata Languages aAnd Programming. 2006. 1-12.[doi:10. 1007/11787006_1]
    [84] Dwork C, Talwar K, Thakurta A, et al. Analyze gauss:Optimal bounds for privacy-preserving principal component analysis. In:Proc. of the Symp. on the Theory of Computing. 2014. 11-20.
    [85] Yu H, Vaidya J, Jiang X, et al. Privacy-preserving SVM classification on vertically partitioned data. In:Proc. of the Knowledge Discovery and Data Mining. 2006. 647-656.[doi:10.1007/11731139_74]
    [86] Hardt M, Price E. The noisy power method:A meta algorithm with applications. In:Proc. of the Neural Information Processing Systems. 2014. 2861-2869.
    [87] Abadi M, Chu A, Goodfellow IJ, et al. Deep learning with differential privacy. In:Proc. of the Computer and Communications Security. 2016. 308-318.[doi:10.1145/2976749.2978318]
    [88] Song S, Chaudhuri K, Sarwate AD, et al. Stochastic gradient descent with differentially private updates. In:Proc. of the IEEE Global Conf. on Signal and Information Processing. 2013. 245-248.[doi:10.1109/GlobalSIP.2013.6736861]
    [89] Geyer RC, Klein T, Nabi M, et al. Differentially private federated learning:A client level perspective. arXiv:1712.07557v2, 2017.
    [90] Chaudhuri K, Sarwate AD, Sinha K, et al. A near-optimal algorithm for differentially-private principal components. Journal of Machine Learning Research, 2013,14(1):2905-2943.[doi:10.1016/j.robot.2013.06.001]
    [91] Chaudhuri K, Monteleoni C, Sarwate AD, et al. Differentially private empirical risk minimization. Journal of Machine Learning Research, 2011,12(3):1069-1109.[doi:10.1109/MIS.2011.2]
    [92] Kung S. Compressive privacy:From information/estimation theory to machine learning. IEEE Signal Processing Magazine, 2017,34(1):94-112.[doi:10.1109/MSP.2016.2616720]
    [93] Hitaj B, Ateniese G, Perezcruz F, et al. Deep models under the GAN:Information leakage from collaborative deep learning. In:Proc. of the Computer and Communications Security. 2017. 603-618.[doi:10.1145/3133956.3134012]
    [94] Papernot N, Abadi M, Erlingsson U, et al. Semi-supervised knowledge transfer for deep learning from private training data. In:Proc. of the Int'l Conf. on Learning Representations. 2017.
    [95] Ohrimenko O, Schuster F, Fournet C, et al. Oblivious multi-party machine learning on trusted processors. In:Proc. of the Usenix Security Symp. 2016. 619-636.
    [96] Zyskind G, Nathan O, Pentland A, et al. Decentralizing privacy:Using blockchain to protect personal data. In:Proc. of the IEEE Symp. on Security and Privacy. 2015. 180-184.[doi:10.1109/SPW.2015.27]
    [97] Outchakoucht A, Essamaali H, Leroy JP, et al. Dynamic access control policy based on blockchain and machine learning for the Internet of things. Int'l Journal of Advanced Computer Science and Applications, 2017,8(7):417-424.[doi:10.14569/IJACSA.2017. 080757]
    附中文参考文献:
    [9] 宋蕾,马春光,段广晗.机器学习安全及隐私保护研究进展.网络与信息安全学报,2018,4(8):1-11.[doi:10.11959/j.issn.2096-109x. 2018067]
    [11] 李盼,赵文涛,刘强,等.机器学习安全性问题及其防御技术研究综述.计算机科学与探索,2018,12(2):171-184.[doi:10.3778/j.issn. 1673-9418.1708038]
    相似文献
    引证文献
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

李欣姣,吴国伟,姚琳,张伟哲,张宾.机器学习安全攻击与防御机制研究进展和未来挑战.软件学报,2021,32(2):406-423

复制
分享
文章指标
  • 点击次数:5021
  • 下载次数: 11091
  • HTML阅读次数: 6015
  • 引用次数: 0
历史
  • 收稿日期:2019-08-12
  • 最后修改日期:2019-12-01
  • 在线发布日期: 2020-10-12
  • 出版日期: 2021-02-06
文章二维码
友情链接:中国科学院软件研究所中国计算机学会中国科学院国家自然科学基金委员会CCF数字图书馆Int'l J of Softw Info计算机系统应用
您是第19987331位访问者
版权所有：中国科学院软件研究所 京ICP备05046678号-3
地址：北京市海淀区中关村南四街4号,邮政编码：100190
电话：010-62562563 传真：010-62562533 Email：jos@iscas.ac.cn
技术支持：北京勤云科技发展有限公司

京公网安备 11040202500063号