面向海量高清视频数据的高性能分布式存储系统
作者:
基金项目:

国家重点研发计划(2016YFB1000202)


High-Performance Distributed Storage System for Large-Scale High-Definition Video Data
Author:
Fund Project:

National Key Research and Development Program of China (2016YFB1000202)

  • 摘要
  • | |
  • 访问统计
  • |
  • 参考文献 [17]
  • |
  • 相似文献 [20]
  • | | |
  • 文章评论
    摘要:

    通过对视频监控数据的特点和传统存储方案进行分析,提出一种高性能分布式存储系统解决方案.不同于传统的基于文件存储的方式,设计了一种逻辑卷结构,将非结构化的视频流数据以此结构进行组织并直接写入RAW磁盘设备,解决了传统存储方案中随机磁盘读写和磁盘碎片导致存储性能下降的问题.该方案将元数据组织为两级索引结构,分别由状态管理器和存储服务器管理,极大地减少了状态管理器需要管理元数据的数量,消除了性能瓶颈,并提供了精确到秒级的检索精度.此外,该方案灵活的存储服务器分组策略和组内互备关系使得存储系统具备容错能力和线性扩展能力.系统测试结果表明,该方案在成本低廉的PC服务器上实现了单台服务器可同时记录400路1080P视频流,写入速度是本地文件系统的2.5倍.

    Abstract:

    This paper presents a high-performance distributed storage solution for video surveillance data via analyzing the characteristics of video surveillance data and conventional file storage solutions. This design proposes a logical volume structure rather than file-based storage to efficiently organize unstructured video stream data. The scheme hence directly writes these stream data into raw disk devices, to address the problem of storage performance decrease caused by the random access and disk fragmentation in traditional storage systems. A two-stage index strategy is also implemented to manage metadata by the state manager and storage servers, which significantly reduces the amount of metadata managed by the state manager, eliminates the performance bottlenecks, and provides the second-level video retrieval accuracy. Moreover, the design has the salient features of fault tolerance and linear scaling abilities with the help of the flexible storage server grouping policy and the mutual backup relationship in a storage group. Experimental results show that the solution can simultaneously record 400 ways of 1080P video streams with a single low-cost PC server, and the system's average write speed is 2.5 times faster compared with the local file systems.

    参考文献
    [1] Wang YJ, Sun WD, Zhou S, Pei XQ, Li XY. Key technologies of distributed storage for cloud computing. Ruan Jian Xue Bao/Journal of Software, 2012,23(4):962-986(in Chinese with English abstract). http://www.jos.org.cn/1000-9825/4175.htm[doi:10. 3724/SP.J.1001.2012.04175]
    [2] He J. Study on key technique in video surveillance storage system based on IP-SAN[MS. Thesis]. Shanghai:Shanghai Jiaotong University, 2011(in Chinese with English abastract).
    [3] Tang JX. The software design of storage subsystem for network video surveillance system[MS. Thesis]. Hangzhou:Zhejiang University, 2013(in Chinese with English abastract).
    [4] Jiang M, Niu ZY, Zhang SP. Design and implementation of video surveillance storage system. Computer Engineering and Design, 2014,35(12):4195-4201(in Chinese with English abstract).[doi:10.3969/j.issn.1000-7024.2014.12.027]
    [5] Sun ZZ, Zhang QX, Tan YA, Li YZ. Ripple-RAID:A high-performance and energy-efficient RAID for continuous data storage. Ruan Jian Xue Bao/Journal of Software, 2015,26(7):1824-1839(in Chinese with English abstract). http://www.jos.org.cn/1000-9825/4606.htm[doi:10.13328/j.cnki.jos.004606]
    [6] Wu JY, Gu Y, Ju DP, Wang DS. THNVR:Distributed large-scale surveillance video storage system. Computer Engineering and Applications, 2009,45(31):56-59(in Chinese with English abstract).[doi:10.3778/j.issn.1002-8331.2009.31.018]
    [7] Lin CF, Yuan SM, Leu MC, Tsai CT. A framework for scalable cloud video recorder system in surveillance environment. In:Proc. of the Ubiquitous Intelligence & Computing and 9th Int'l Conf. on Autonomic & Trusted Computing (UIC/ATC). Fukuoka:IEEE, 2012. 655-660.[doi:10.1109/UIC-ATC.2012.72]
    [8] Duan HC, Zhan WH, Min GY, Guo H, Luo SM. A high-performance distributed file system for large-scale concurrent HD video streams. Concurrency and Computation:Practice and Experience, 2015,27(13):3510-3522.[doi:10.1002/cpe.3528]
    [9] Deng P, Li MY, He C. Research on namenode single point of fault solution. Computer Engineering, 2012,38(21):40-44(in Chinese with English abstract).[doi:10.3969/j.issn.1000-3428.2012.21.011]
    附中文参考文献:
    [1] 王意洁,孙伟东,周松,裴晓强,李小勇.云计算环境下的分布存储关键技术.软件学报,2012,23(4):962-986. http://www.jos.org.cn/1000-9825/4175.htm[doi:10.3724/SP.J.1001.2012.04175]
    [2] 何炬.基于IP-SAN的视频监控存储系统关键技术研究[硕士学位论文].上海:上海交通大学,2011.
    [3] 汤家兴.网络视频监控系统存储子系统软件设计[硕士学位论文].杭州:浙江大学,2013.
    [4] 江冕,牛中盈,张淑萍.视频监控存储系统的设计与实现.计算机工程与设计,2014,35(12):4195-4201.[doi:10.3969/j.issn.1000-7024.2014.12.027]
    [5] 孙志卓,张全新,谭毓安,李元章.Ripple-RAID:一种面向连续数据存储的高效能盘阵.软件学报,2015,26(7):1824-1839. http://www.jos.org.cn/1000-9825/4606.htm[doi:10.13328/j.cnki.jos.004606]
    [6] 邬建元,顾瑜,鞠大鹏,江东升.分布式大规模监控视频存储系统THNVR.计算机工程与应用,2009,45(31):56-59.[doi:10.3778/j. issn.1002-8331.2009.31.018]
    [9] 邓鹏,李枚毅,何诚.Namenode单点故障解决方案研究.计算机工程,2012,38(21):40-44.[doi:10.3969/j.issn.1000-3428.2012.21. 011]
    引证文献
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

操顺德,华宇,冯丹,孙园园,左鹏飞.面向海量高清视频数据的高性能分布式存储系统.软件学报,2017,28(8):1999-2009

复制
分享
文章指标
  • 点击次数:4762
  • 下载次数: 8882
  • HTML阅读次数: 3365
  • 引用次数: 0
历史
  • 收稿日期:2016-08-11
  • 最后修改日期:2016-09-21
  • 在线发布日期: 2017-08-15
文章二维码
您是第20544918位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62562563 传真:010-62562533 Email:jos@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号