分布式数据流上的高性能分发策略
作者:
中图分类号:

TP311

基金项目:

国家高技术研究发展计划(863)(2015AA015307);国家自然科学基金(61232002,61332006,61572194)


High-Performance Data Distribution Algorithm on Distributed Stream Systems
Author:
  • 摘要
  • | |
  • 访问统计
  • |
  • 参考文献 [31]
  • |
  • 相似文献 [20]
  • | | |
  • 文章评论
    摘要:

    随着大数据应用的普及,高效可扩展的数据流操作在实时分析处理中扮演着越来越重要的角色.分布式并行处理架构是应对大流量、低延时数据流处理任务的一种有效解决方案.然而在Key-based分组并行处理中,由于数据的倾斜分布及数据流本身的实时、动态和数据规模不可预知等特性,使得数据流分布并行处理系统存在持续且动态负载不均衡现象,这会造成系统时效性降低、硬件资源浪费等问题.现有的研究工作处理均衡负载有两种方案:(1)基于key粒度的迁移,使得并行处理节点负载达到均衡;(2)基于元组粒度级别的拆分,采用随机分发使系统均衡.前者将系统调整至给定的均衡容忍范围内,类似于一维装箱的NP问题;后者对key的拆分势必带来新的为维护Key-based操作的正确性而增加的额外代价,如内存及网络通信成本.综合两种方法,提出对key按需拆分、尽量合并的方法,通过轻量级均衡调整算法以及保证Key-based操作特性的拆分方法,使系统既能达到后者的均衡,又能减少细粒度均衡所带来的额外代价.

    Abstract:

    Along with the popularization of big data applications, scalable and efficient stream join processing plays a more important role in online real-time analysis. The distributed parallel processing framework provides an effective solution which facilitates processing of massive data stream with low latency. For Key-based calculations, data skewness and inherent features of stream data, such as real-time, dynamics and unpredictability on data volume, lead to load imbalance to distributed processing systems. Such phenomenon can produce poor performance and waste hardware resources. There have been two solutions to load imbalance:1) Key-based migration scheme that keeps balance among parallel processing nodes; 2) tuple-based partitioning scheme that distributes data randomly to achieve load balance. The former scheme adjusts system to the defined equilibrium range, which resembles the one-dimensional packing problem. And the latter maintains the accuracy of Key-based operations, which certainly incurs additional memory cost and network communication cost. This paper presents a novel parallel processing scheme that combines both Key-based and tuple-based schemes to partition keys on demand. The proposed scheme adopts a lightweight load balance algorithm and a partitioning scheme which retains the characteristics of Key-based operations, thus realizing the load balance of tuple-base strategy while reducing the additional cost of fine-grained balance.

    参考文献
    [1] Gong XQ, Jin CQ, Wang XL, Zhang R, Zhou AY. Data-Intensive science and engineering:Requirements and challenges. Chinese Journal of Computers, 2012,35(8):1-16(in Chinese with English abstract).
    [2] Li JZ, Liu XM. An important aspect of big data:Data usability. Journal of Computer Research and Development, 2013,50(6):1147-1162(in Chinese with English abstract).
    [3] Qian Z, He Y, Su C, Wu ZJ, Zhu HY. TimeStream:Reliable stream computation in the cloud. In:Proc. of the ACM European Conf. on Computer Systems. 2013. 1-14.[doi:10.1145/2465351.2465353]
    [4] Zhou Y, Ooi BC, Tan KL. Dynamic load management for distributed continuous query systems. In:Proc. of the IEEE Computer Society. 2014. 322-323.[doi:10.1109/ICDE.2005.54]
    [5] Zhu Y, Rundensteiner EA, Heineman GT. Dynamic plan migration for continuous queries over data streams. In:Proc. of the ACM SIGMOD Int'l Conf. on Management of Data. Paris, 2015. 431-442.[doi:10.1145/1007568.1007617]
    [6] Zhou YL, Ooi BC, Tan KL, Wu J. Efficient dynamic operator placement in a locally distributed continuous query system. LNCS 4275, 2006. 54-71.[doi:10.1007/11914853_5]
    [7] Fernandez RC, Migliavacca M, Kalyvianaki E, Pietzuch P. Integrating scale out and fault tolerance in stream processing using operator state management. In:Proc. of the 2013 ACM SIGMOD Int'l Conf. on Management of Data. 2013. 725-736.[doi:10. 1145/2463676.2465282]
    [8] Jin CQ, Qian WN, Zhou AY. Analysis and management of streaming data:A survey. Ruan Jian Xue Bao/Journal of Software, 2004, 15(08):1172-1181(in Chinese with English abstract). http://www.jos.org.cn/1000-9825/15/1172.htm
    [9] Xu Y, Kostamaa P, Zhou X, Chen L. Handling data skew in parallel joins in shared-nothing systems. In:Proc. of the ACM SIGMOD Int'l Conf. on Management of Data. 2008. 1043-1052.[doi:10.1145/1376616.1376720]
    [10] Vitorovic A, Elseidy M, Koch C. Load balancing and skew resilience for parallel joins. In:Proc. of the IEEE Int'l Conf. on Data Engineering. IEEE, 2016. 313-324.[doi:10.1109/ICDE.2016.7498250]
    [11] Kwon YC, Balazinska M, Howe B, Rolia J. SkewTune:Mitigating skew in mapreduce applications. In:Proc. of the ACM SIGMOD Int'l Conf. on Management of Data. 2012. 25-36.[doi:10.1145/2213836.2213840]
    [12] Gufler B, Augsten N, Reiser A, kemper A. Load balancing in MapReduce based on scalable cardinality estimates. In:Proc. of the IEEE Int'l Conf. on Data Engineering. 2012. 522-533.[doi:10.1109/ICDE.2012.58]
    [13] Xing Y, Zdonik S, Hwang JH. Dynamic load distribution in the Borealis stream processor. In:Proc. of the Int'l Conf. on Data Engineering. IEEE, 2005. 791-802.[doi:10.1109/ICDE.2005.53]
    [14] Xing Y, Hwang JH, Cetintemel U, Zdonik S. Providing resiliency to load variations in distributed stream processing. In:Proc. of the Int'l Conf. on Very Large Data Bases. ACM Press, 2006. 775-786.
    [15] Abadi DJ, Ahmad Y, Balazinska M, et al. The design of the Borealis stream processing engine. In:Proc. of the 2005 CIDR Conf., 2005. 277-289.
    [16] Fang J, Zhang R, Fu TZJ, Zhang ZJ, Zhou AY, Zhu JH. Parallel stream processing against workload skewness and variance. arXiv preprint arXiv:1610. 05121, 2016.
    [17] Shah MA, Hellerstein JM, Chandrasekaran S, Franklin MJ. Flux:An adaptive partitioning operator for continuous query systems. In:Proc. of the Int'l Conf. on Data Engineering. 2003. 25-36.[doi:10.1109/ICDE.2003.1260779]
    [18] Gedik B. Partitioning functions for stateful data parallelism in stream processing. Int'l Journal on Very Large Data Bases, 2014, 23(4):517-539.[doi:10.1007/s00778-013-0335-9]
    [19] Lin Q, Ooi BC, Wang Z, Yu C. Scalable distributed stream join processing. In:Proc. of the ACM SIGMOD Int'l Conf. 2015. 811-825.[doi:10.1145/2723372.2746485]
    [20] Nasir MAU, Morales GDF, Garcia-Soriano D, Kourtellis N, Serafini M. The power of both choices:Practical load balancing for distributed stream processing engines. In:Proc. of the IEEE Int'l Conf. on Data Engineering. IEEE, 2015. 137-148.[doi:10.1109/ICDE.2015.7113279]
    [21] Apache storm. http://storm.apache.org/
    [22] Elseidy M, Elguindy A, Vitorovic A, Koch C. Scalable and adaptive online joins. Proc. of the VLDB Endowment, 2014,7(6):441-452.[doi:10.14778/2732279.2732281]
    [23] Nasir MAU, Morales GDF, Kourtellis N, Serafini M. When two choices are not enough:Balancing at scale in distributed stream processing. In:Proc. of 2016 IEEE 32nd Int'l Conf. on Data Engineering, 2016. 589-600.
    [24] Fang JH, Zhang R, Wang XT, Fu TZJ, Zhang ZJ, Zhou AY. Cost-Effective stream join algorithm on cloud system. In:Proc. of the 25th ACM Int'l on Conf. on Information and Knowledge Management. ACM Press, 2016. 1773-1782.[doi:10.1145/2983323. 2983773]
    [25] Okcan A, Riedewald M. Processing theta-joins using MapReduce. In:Proc. of the ACM SIGMOD Int'l Conf. on Management of Data. 2011. 949-960.[doi:10.1145/1989323.1989423]
    [26] Fang JH, Wang XT, Zhang R, Zhou AY. Flexible and adaptive stream join algorithm. In:Proc. of the Asia-Pacific Web Conf. Springer Int'l Publishing, 2016. 3-16.[doi:10.1007/978-3-319-45817-5_1]
    [27] Bruno N, Kwon YC, Wu MC. Advanced join strategies for large-scale distributed computation. Proc. of the VLDBEndowment, 2014,7(13):1484-1495.[doi:10.14778/2733004.2733020]
    [28] The TPC-H benchmark. http://www.tpc.org/tpch
    附中文参考文献:[1] 宫学庆,金澈清,王晓玲,张蓉,周傲英.数据密集型科学与工程:需求和挑战.计算机学报,2012,35(8):1-16.
    [2] 李建中,刘显敏.大数据的一个重要方面:数据可用性.计算机研究与发展,2013,50(6):1147-1162.
    [8] 金澈清,钱卫宁,周傲英.流数据分析与管理综述.软件学报,2004,15(8):1172-1181. http://www.jos.org.cn/1000-9825/15/1172.htm
    引证文献
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

房俊华,王晓桐,张蓉,周傲英.分布式数据流上的高性能分发策略.软件学报,2017,28(3):563-578

复制
分享
文章指标
  • 点击次数:2691
  • 下载次数: 5490
  • HTML阅读次数: 1466
  • 引用次数: 0
历史
  • 收稿日期:2016-07-31
  • 最后修改日期:2016-09-14
  • 在线发布日期: 2018-06-06
文章二维码
您是第19877004位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62562563 传真:010-62562533 Email:jos@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号