微信服务号

微信订阅号

2025-4-8- 4
Home > Archive>Volume 28, Issue 8, 2017 >1952-1967. DOI:10.13328/j.cnki.jos.005199
PDF HTML XML Export Cite reminder
Adaptive Load Balancing Strategy for File-System Metadata Service
Author:
Affiliation:

Fund Project:

Science and Technology Project of Guangdong Province (2014B010114002, 2015B010108004)

  • Article
    • | |
  • Metrics
    • |
  • Reference [26]
    • |
  • Related
    • |
  • Cited by
    • | |
  • Comments
    Abstract:

    With the advent of big data era, global storage is experiencing an explosive growth. Traditional storage systems have several drawbacks in storage performance, storage capacity, data security and device cost. In order to handle large amount of data, the storage technology for cloud computing platform has undergone rapid development in the recent years, and become an important tool to deal with big data. This paper analyzes the shortcomings of metadata management of certain distributed file system and proposes an adaptive metadata load balancing mechanism. First, a real-time server performance evaluation model is introduced. Next, a period adaptive mechanism based on change of server load is built. Finally, an adaptive load balancing algorithm based on server performance is proposed. Experimental results demonstrate the practicability, availability and stability of the new mechanism.

    Reference
    [1] Jiang T, Hou R, Zhang LX, Zhang K, Chen LC, Chen MY, Sun N. Micro-Architectural characterization of desktop cloud workloads. In:Proc. of the 2012 IEEE Int'l Symp. on Workload Characterization (ⅡSWC). 2012. 131-140.[doi:10.1109/ⅡSWC.2012. 6402917]
    [2] Wang F, Nelson M, Oral S, Atchley S, Weil S, Settlemyer BW, Caldwell B, Hill J. Performance and scalability evaluation of the Ceph parallel file system. In:Proc. of the 8th Parallel Data Storage Workshop. New York, 2013. 14-19.[doi:10.1145/2538542. 2538562]
    [3] Ghemawat S, Gobioff H, Leung ST. The Google file system. In:Proc. of the 19th ACM Symp. on Operating Systems Principles (SOSP 2003). New York, 2003. 29-43.
    [4] Shvachko K, Kuang H, Radia S, Chansler R. The hadoop distributed file system. In:Proc. of 2010 IEEE the 26th Symp. on Mass Storage Systems & Technologies. 2010(11):1-10.[doi:10.1109/MSST.2010.5496972]
    [5] Weil SA, Brandt SA, Miller EL, Long DDE, Maltzahn C. Ceph:A scalable, high-performance distributed file system. In:Proc. of the 7th Symp. on Operating Systems Design and Implementation. Washington, 2006. 307-320.
    [6] Liu JL, Zhang YL, Yang L, Guo MY, Liu ZJ, Xu L. SAC:Exploiting stable set model to enhance cache files. Journal of Computer Science & Technology, 2014,29(2):293-302.[doi:10.1007/s11390-014-1431-z]
    [7] Kim T, Noh SH. pNFS for everyone:An empirical study of a low-cost, highly scalable networked storage. Int'l Journal of Computer Science & Network Security, 2014,14(3):52-59.
    [8] Sun Microsystems, Inc. Lustre file system:High-Performance storage architecture and scalable cluster file system. 2008. https://www.sun.com/offers/docs/LustreFileSystem.pdf
    [9] Rogers GL, Hanley J, Mohr R. Data management practices on large-scale lustre scratch file systems. In:Proc. of the 14th Conf. on Extreme Science and Engineering Discovery Environment (XSEDE 2014). Atlanta, 2014. 1-6.[doi:10.1145/2616498.2616545]
    [10] Thomson A, Abadi DJ. CalvinFS:Consistent WAN replication and scalable metadata management for distributed file systems. In:Proc. of the 13th USENIX Conf. on File and Storage Technologies (FAST 2015). Santa Clara, 2015. 1-14.
    [11] Thomson A, Diamond T, Weng SC, Ren K, Shao P, Abadi DJ. Calvin:Fast distributed transactions for partitioned database systems. In:Proc. of ACM SIGMOD Int'l Conf. on Management of Data (SIGMOD 2012). Scottsdale, 2012. 1-12.[doi:10.1145/2213836. 2213838]
    [12] Yuan J, Zhan Y, Jannen W, Pandey P, Akshintala A, Chandnani K, Deo P, Kasheff Z, Walsh L, Bender MA, Farach-Colton M, Johnson R, Kuszmaul BC, Porter DE. Optimizing every operation in a write-optimized file system. In:Proc. of the 14th USENIX Conf. on File and Storage Technologies (FAST 2016). Santa Clara, 2016. 1-14.
    [13] Jannen W, Yuan J, Zhan Y, Akshintala A, Esmet J, Jiao Y, Mittal A, Pandey P, Reddy P, Walsh L, Bender M, Farach-Colton M, Johnson R, Kuszmaul BC, Porter DE. BetrFS:A right-optimized write-optimized file system. In:Proc. of the 13th USENIX Conf. on File and Storage Technologies (FAST 2015). Santa Clara, 2015. 301-315.
    [14] Esmet J, Bender MA, Farach-Colton M, Kuszmaul BC. The TokuFS streaming file system. In:Proc. of the USENIX Conf. on Hot Topics in Storage & File Systems. 2012. 1-5.
    [15] Liu J, Zhang JW, Shao BQ, Dong HQ, Liu ZJ, Xu L. Metadata server clustering system for EB-scale storage. Zhong Guo Ke Xue/Science China, 2015,45(6):721-738(in Chinese with English abstract).[doi:10.1360/N112014-00330]
    [16] Liu Z, Zhou XM. A metadata management method based on directory path. Ruan Jian Xue Bao/Journal of Software, 2007,18(2):236-245(in Chinese with English abstract). http://www.jos.org.cn/1000-9825/18/236.htm[doi:10.1360/jos180236]
    [17] Chen T, Xiao N, Liu F. Adaptive metadata load balancing for object storage systems. Ruan Jian Xue Bao/Journal of Software, 2013, 24(2):331-342(in Chinese with English abstract). http://www.jos.org.cn/1000-9825/4177.htm[doi:10.3724/SP.J.1001.2013. 04177]
    [18] Rodeh O, Teperman A. zFS-A scalable distributed file system using object disks. In:Proc. of the IEEE Nasa Goddard Conf. on Mass Storage Systems & Technologies. 2003. 207-218.
    [19] Zhang X, Li L, Wang S, Yang F. Improved selective randomized load balancing in mesh networks. ETRI Journal, 2007,29(2):255-257.[doi:10.4218/etrij.07.0206.0215]
    [20] Zhang JL, Qian W, Xu XH, Wan J, Yin YY, Ren YJ. WLBS:A weight-based metadata server cluster load balancing strategy. Int'l Journal of Advancements in Computing Technology, 2012,4(1):77-85.[doi:10.4156/ijact.vol4.issue1.9]
    [21] Hua Y, Zhu Y, Jiang H, Feng D, Tian L. Supporting scalable and adaptive metadata management in ultralarge-scale file systems. IEEE Trans. on Parallel and Distributed Systems, 2011,22(4):580-593.[doi:10.1109/TPDS.2010.116]
    [22] Li B, He Y, Xu K. Distributed metadata management scheme in cloud computing. In:Proc. of the 6th Int'l Conf. on IEEE Pervasive Computing and Applications (ICPCA). 2011. 32-38.[doi:10.1109/ICPCA.2011.6106475]
    附中文参考文献:
    [15] 刘健,张军伟,邵冰清,董欢庆,刘振军,许鲁.支持EB级存储的元数据服务器集群系统.中国科学:信息科学,2015,45(6):721-738.[doi:10.1360/N112014-00330]
    [16] 刘仲,周兴铭.基于目录路径的元数据管理方法.软件学报,2007,18(2):236-245. http://www.jos.org.cn/1000-9825/18/236.htm[doi:10.1360/jos180236]
    [17] 陈涛,肖侬,刘芳.对象存储系统中自适应的元数据负载均衡机制.软件学报,2013,24(2):331-342. http://www.jos.org.cn/1000-9825/4177.htm[doi:10.3724/SP.J.1001.2013.04177]
    Related
    Cited by
Get Citation

佘楚玉,温武少,肖扬,刘育擘,贾殷.一种自适应文件系统元数据服务负载均衡策略.软件学报,2017,28(8):1952-1967

Copy
Share
Article Metrics
  • Abstract:
  • PDF:
  • HTML:
  • Cited by:
History
  • Received:June 17,2016
  • Revised:September 21,2016
  • Online: August 15,2017
Links:Institute of Software, CASChina Computer FederationCASNSFC
You are the first2033794Visitors
Copyright: Institute of Software, Chinese Academy of Sciences Beijing ICP No. 05046678-4
Address：4# South Fourth Street, Zhong Guan Cun, Beijing 100190,Postal Code：100190
Phone：010-62562563 Fax：010-62562533 Email：jos@iscas.ac.cn
Technical Support：Beijing Qinyun Technology Development Co., Ltd.

Beijing Public Network Security No. 11040202500063