微信服务号

微信订阅号

2025年8月4日 21:08 星期一
首页 > 过刊浏览>2017年第28卷第8期 >1952-1967. DOI:10.13328/j.cnki.jos.005199
PDF HTML阅读 XML下载 导出引用 引用提醒
一种自适应文件系统元数据服务负载均衡策略
作者:
基金项目:

广东省科技计划（2014B010114002，2015B010108004）


Adaptive Load Balancing Strategy for File-System Metadata Service
Author:
Fund Project:

Science and Technology Project of Guangdong Province (2014B010114002, 2015B010108004)

  • 摘要
    • | |
  • 访问统计
    • |
  • 参考文献 [26]
    • |
  • 相似文献 [20]
    • | | |
  • 文章评论
    摘要:

    随着大数据时代的到来，全球信息存储量呈现爆发式的增长，传统的存储系统在存储性能、存储容量、数据可靠性和成本等方面存在诸多不足.近年来，以云计算平台为依托的存储技术得到了飞速发展，成为处理海量数据的重要工具.针对分布式文件系统元数据管理的问题，提出了一种自适应元数据服务负载均衡策略.该策略主要包括以下3点内容：介绍了一种实时的元数据服务器的性能评价模型；提出了一种基于服务器负载变化的检测周期自适应调整机制；提出了一种基于元数据服务器性能指标的自适应负载均衡算法.实验结果证明了该方法的可行性、有效性和稳定性.

    Abstract:

    With the advent of big data era, global storage is experiencing an explosive growth. Traditional storage systems have several drawbacks in storage performance, storage capacity, data security and device cost. In order to handle large amount of data, the storage technology for cloud computing platform has undergone rapid development in the recent years, and become an important tool to deal with big data. This paper analyzes the shortcomings of metadata management of certain distributed file system and proposes an adaptive metadata load balancing mechanism. First, a real-time server performance evaluation model is introduced. Next, a period adaptive mechanism based on change of server load is built. Finally, an adaptive load balancing algorithm based on server performance is proposed. Experimental results demonstrate the practicability, availability and stability of the new mechanism.

    参考文献
    [1] Jiang T, Hou R, Zhang LX, Zhang K, Chen LC, Chen MY, Sun N. Micro-Architectural characterization of desktop cloud workloads. In:Proc. of the 2012 IEEE Int'l Symp. on Workload Characterization (ⅡSWC). 2012. 131-140.[doi:10.1109/ⅡSWC.2012. 6402917]
    [2] Wang F, Nelson M, Oral S, Atchley S, Weil S, Settlemyer BW, Caldwell B, Hill J. Performance and scalability evaluation of the Ceph parallel file system. In:Proc. of the 8th Parallel Data Storage Workshop. New York, 2013. 14-19.[doi:10.1145/2538542. 2538562]
    [3] Ghemawat S, Gobioff H, Leung ST. The Google file system. In:Proc. of the 19th ACM Symp. on Operating Systems Principles (SOSP 2003). New York, 2003. 29-43.
    [4] Shvachko K, Kuang H, Radia S, Chansler R. The hadoop distributed file system. In:Proc. of 2010 IEEE the 26th Symp. on Mass Storage Systems & Technologies. 2010(11):1-10.[doi:10.1109/MSST.2010.5496972]
    [5] Weil SA, Brandt SA, Miller EL, Long DDE, Maltzahn C. Ceph:A scalable, high-performance distributed file system. In:Proc. of the 7th Symp. on Operating Systems Design and Implementation. Washington, 2006. 307-320.
    [6] Liu JL, Zhang YL, Yang L, Guo MY, Liu ZJ, Xu L. SAC:Exploiting stable set model to enhance cache files. Journal of Computer Science & Technology, 2014,29(2):293-302.[doi:10.1007/s11390-014-1431-z]
    [7] Kim T, Noh SH. pNFS for everyone:An empirical study of a low-cost, highly scalable networked storage. Int'l Journal of Computer Science & Network Security, 2014,14(3):52-59.
    [8] Sun Microsystems, Inc. Lustre file system:High-Performance storage architecture and scalable cluster file system. 2008. https://www.sun.com/offers/docs/LustreFileSystem.pdf
    [9] Rogers GL, Hanley J, Mohr R. Data management practices on large-scale lustre scratch file systems. In:Proc. of the 14th Conf. on Extreme Science and Engineering Discovery Environment (XSEDE 2014). Atlanta, 2014. 1-6.[doi:10.1145/2616498.2616545]
    [10] Thomson A, Abadi DJ. CalvinFS:Consistent WAN replication and scalable metadata management for distributed file systems. In:Proc. of the 13th USENIX Conf. on File and Storage Technologies (FAST 2015). Santa Clara, 2015. 1-14.
    [11] Thomson A, Diamond T, Weng SC, Ren K, Shao P, Abadi DJ. Calvin:Fast distributed transactions for partitioned database systems. In:Proc. of ACM SIGMOD Int'l Conf. on Management of Data (SIGMOD 2012). Scottsdale, 2012. 1-12.[doi:10.1145/2213836. 2213838]
    [12] Yuan J, Zhan Y, Jannen W, Pandey P, Akshintala A, Chandnani K, Deo P, Kasheff Z, Walsh L, Bender MA, Farach-Colton M, Johnson R, Kuszmaul BC, Porter DE. Optimizing every operation in a write-optimized file system. In:Proc. of the 14th USENIX Conf. on File and Storage Technologies (FAST 2016). Santa Clara, 2016. 1-14.
    [13] Jannen W, Yuan J, Zhan Y, Akshintala A, Esmet J, Jiao Y, Mittal A, Pandey P, Reddy P, Walsh L, Bender M, Farach-Colton M, Johnson R, Kuszmaul BC, Porter DE. BetrFS:A right-optimized write-optimized file system. In:Proc. of the 13th USENIX Conf. on File and Storage Technologies (FAST 2015). Santa Clara, 2015. 301-315.
    [14] Esmet J, Bender MA, Farach-Colton M, Kuszmaul BC. The TokuFS streaming file system. In:Proc. of the USENIX Conf. on Hot Topics in Storage & File Systems. 2012. 1-5.
    [15] Liu J, Zhang JW, Shao BQ, Dong HQ, Liu ZJ, Xu L. Metadata server clustering system for EB-scale storage. Zhong Guo Ke Xue/Science China, 2015,45(6):721-738(in Chinese with English abstract).[doi:10.1360/N112014-00330]
    [16] Liu Z, Zhou XM. A metadata management method based on directory path. Ruan Jian Xue Bao/Journal of Software, 2007,18(2):236-245(in Chinese with English abstract). http://www.jos.org.cn/1000-9825/18/236.htm[doi:10.1360/jos180236]
    [17] Chen T, Xiao N, Liu F. Adaptive metadata load balancing for object storage systems. Ruan Jian Xue Bao/Journal of Software, 2013, 24(2):331-342(in Chinese with English abstract). http://www.jos.org.cn/1000-9825/4177.htm[doi:10.3724/SP.J.1001.2013. 04177]
    [18] Rodeh O, Teperman A. zFS-A scalable distributed file system using object disks. In:Proc. of the IEEE Nasa Goddard Conf. on Mass Storage Systems & Technologies. 2003. 207-218.
    [19] Zhang X, Li L, Wang S, Yang F. Improved selective randomized load balancing in mesh networks. ETRI Journal, 2007,29(2):255-257.[doi:10.4218/etrij.07.0206.0215]
    [20] Zhang JL, Qian W, Xu XH, Wan J, Yin YY, Ren YJ. WLBS:A weight-based metadata server cluster load balancing strategy. Int'l Journal of Advancements in Computing Technology, 2012,4(1):77-85.[doi:10.4156/ijact.vol4.issue1.9]
    [21] Hua Y, Zhu Y, Jiang H, Feng D, Tian L. Supporting scalable and adaptive metadata management in ultralarge-scale file systems. IEEE Trans. on Parallel and Distributed Systems, 2011,22(4):580-593.[doi:10.1109/TPDS.2010.116]
    [22] Li B, He Y, Xu K. Distributed metadata management scheme in cloud computing. In:Proc. of the 6th Int'l Conf. on IEEE Pervasive Computing and Applications (ICPCA). 2011. 32-38.[doi:10.1109/ICPCA.2011.6106475]
    附中文参考文献:
    [15] 刘健,张军伟,邵冰清,董欢庆,刘振军,许鲁.支持EB级存储的元数据服务器集群系统.中国科学:信息科学,2015,45(6):721-738.[doi:10.1360/N112014-00330]
    [16] 刘仲,周兴铭.基于目录路径的元数据管理方法.软件学报,2007,18(2):236-245. http://www.jos.org.cn/1000-9825/18/236.htm[doi:10.1360/jos180236]
    [17] 陈涛,肖侬,刘芳.对象存储系统中自适应的元数据负载均衡机制.软件学报,2013,24(2):331-342. http://www.jos.org.cn/1000-9825/4177.htm[doi:10.3724/SP.J.1001.2013.04177]
    引证文献
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

佘楚玉,温武少,肖扬,刘育擘,贾殷.一种自适应文件系统元数据服务负载均衡策略.软件学报,2017,28(8):1952-1967

复制
分享
文章指标
  • 点击次数:4423
  • 下载次数: 6496
  • HTML阅读次数: 3725
  • 引用次数: 0
历史
  • 收稿日期:2016-06-17
  • 最后修改日期:2016-09-21
  • 在线发布日期: 2017-08-15
文章二维码
友情链接:中国科学院软件研究所中国计算机学会中国科学院国家自然科学基金委员会CCF数字图书馆Int'l J of Softw Info计算机系统应用
您是第20434594位访问者
版权所有：中国科学院软件研究所 京ICP备05046678号-3
地址：北京市海淀区中关村南四街4号,邮政编码：100190
电话：010-62562563 传真：010-62562533 Email：jos@iscas.ac.cn
技术支持：北京勤云科技发展有限公司

京公网安备 11040202500063号