微信服务号

微信订阅号

2025年4月26日 21:00 星期六
首页 > 过刊浏览>2018年第29卷第6期 >1770-1791. DOI:10.13328/j.cnki.jos.005557
PDF HTML阅读 XML下载 导出引用 引用提醒
MapReduce与Spark用于大数据分析之比较
作者:
作者简介:

吴信东(1963-),男,安徽枞阳人,博士,教授,博士生导师,主要研究领域为数据挖掘,大数据分析,知识库系统,万维网信息探索;嵇圣硙(1994-),男,硕士生,主要研究领域为数据挖掘,分布式数据库.

通讯作者:

吴信东,E-mail:xwu@hfut.edu.cn

基金项目:

国家重点研发计划（2016YFB1000901）；国家自然科学基金（91746209）；教育部创新团队项目（IRT17R3）


Comparative Study on MapReduce and Spark for Big Data Analytics
Author:
Fund Project:

National Key Researh and Development Program of China (2016YFB1000901); National Natural ScienceFoundation of China (91746209); Program for Changjiang Scholars and Innovative Research Team in University (PCSIRT) of theMinistry of Education (IRT17R3)

  • 摘要
    • | |
  • 访问统计
    • |
  • 参考文献 [50]
    • |
  • 相似文献
    • |
  • 引证文献
    • | |
  • 文章评论
    摘要:

    评述了MapReduce与Spark两种大数据计算算法和架构，从背景、原理以及应用场景进行分析和比较，并对两种算法各自优点以及相应的限制做出了总结.当处理非迭代问题时，MapReduce凭借其自身的任务调度策略和shuffle机制，在中间数据传输数量以及文件数目方面的性能要优于Spark；而在处理迭代问题和一些低延迟问题时，Spark可以根据数据之间的依赖关系对任务进行更合理的划分，相较于MapReduce，有效地减少了中间数据传输数量与同步次数，提高了系统的运行效率.

    Abstract:

    This paper reviews two state-of-the-art algorithmic architectures, MapReduce and Spark, and compares them from their backgrounds, principles and application scenarios. The advantages and their corresponding limitations of these two algorithms are summarized. When dealing with non-iterative problems, MapReduce, by virtue of its task scheduling strategy and shuffle mechanisms, performs better than Spark in terms of intermediate data transfers and number of files. Spark can be used to deal with iterative problems and low latency issues, as it divides a computing task according to the dependencies between the data and the task. Compared with MapReduce, Spark can effectively reduce the number of intermediate data transmissions and the number of synchronizations, and improve the running efficiency of computing systems.

    参考文献
    [1] Gordon K. What is big data? Itnow, 2013,55(3):12-13.
    [2] Dean J, Ghemawat S. MapReduce:Simplified data processing on large clusters. In:Proc. of the 6th Conf. on Symp. on Opearting Systems Design and Implementation. San Francisco:USENIX Association Berkeley, 2004. 137-149.
    [3] Rao BT, Sridevi N, Reddy VK, Reddy L. Performance issues of heterogeneous hadoop clusters in cloud computing. Global Journal of Computer Science and Technology, 2011,6(8):1-6.
    [4] Zaharia M, Chowdhury M, Franklin MJ, Shenker S, Stoica I. Spark:Cluster computing with working sets. HotCloud, 2010,10(10-10):1-7.
    [5] Dean J. Experiences with MapReduce, an abstraction for large-scale computation. In:Proc. of the 15th Int'l Conf. on Parallel Architectures and Compilation Techniques. Washington:ACM Press, 2006. 1-1.
    [6] Srirama SN, Jakovits P, Vainikko E. Adapting scientific computing problems to clouds using MapReduce. Future Generation Computer Systems, 2012,28(1):184-192.[doi:10.1016/j.future.2011.05.025]
    [7] Elghandour I, Aboulnaga A. ReStore:Reusing results of MapReduce jobs in pig. In:Proc. of the 2012 ACM SIGMOD Int'l Conf. on Management of Data. Scottsdale:ACM Press, 2012. 701-704.[doi:10.1145/2213836.2213937]
    [8] Pansare N, Borkar VR, Jermaine C, Condie T. Online aggregation for large MapReduce jobs. Proc. of the Vldb Endowment, 2012, 4(11):1135-1145.
    [9] Zaharia M, Borthakur D, Sarma JS, Elmeleegy K, Shenker S, Stoica I. Job scheduling for multi-user MapReduce clusters. Technical Report UCB/EECS-2009-55. Berkeley:EECS Department, University of California, 2009. 1-18.
    [10] Zaharia M. Job scheduling with the fair and capacity schedulers. In:Proc. of the Hadoop Summit. 2009. 9.
    [11] Nightingale EB, Chen PM, Flinn J. Speculative execution in a distributed file system. ACM SIGOPS Operating Systems Review, 2005,39(5):191-205.[doi:10.1145/1095809.1095829]
    [12] Isard M, Budiu M, Yu Y, Birrell A, Fetterly D. Dryad:Distributed data-parallel programs from sequential building blocks. In:Proc. of the 2nd ACM SIGOPS/EuroSys European Conf. on Computer Systems 2007. Lisbon:ACM Press, 2007. 59-72.[doi:10.1145/1272998.1273005]
    [13] Venner J. Pro Hadoop. Berkeley:Apress, 2009. 1-440.[doi:10.1007/978-1-4302-1943-9]
    [14] Shvachko K, Kuang H, Radia S, Chansler R. The hadoop distributed file system. In:Proc. of the 2010 IEEE 26th Symp. on Mass Storage Systems and Technologies (MSST). Nevada:IEEE, 2010. 1-10.[doi:10.1109/MSST.2010.5496972]
    [15] Vavilapalli VK, Murthy AC, Douglas C, Agarwal S, Konar M, Evans R, Graves T, Lowe J, Shah H, Seth S. Apache hadoop yarn:Yet another resource negotiator. In:Proc. of the 4th Annual Symp. on Cloud Computing. ACM Press, 2013. 1-16.[doi:10.1145/2523616.2523633]
    [16] Wang J, Shang P, Yin J. DRAW:A New Data-gRouping-AWare Data Placement Scheme for Data Intensive Applications with Interest Locality. New York:Springer, 2014. 149-174.[doi:10.1007/978-1-4939-1905-5]
    [17] Amer A, Long DD, Burns RC. Group-Based management of distributed file caches. In:Proc. of the 200222nd Int'l Conf. on Distributed Computing Systems. Vienna:IEEE, 2002. 525-534.[doi:10.1109/ICDCS.2002.1022302]
    [18] Chen Q, Zhang D, Guo M, Deng Q, Guo S. Samr:A self-adaptive MapReduce scheduling algorithm in heterogeneous environment. In:Proc. of the 2010 IEEE 10th Int'l Conf. on Computer and Information Technology (CIT). Bradford:IEEE, 2010. 2736-2743.[doi:10.1109/CIT.2010.458]
    [19] Kwon Y, Balazinska M, Howe B, Rolia J. Skewtune:Mitigating skew in MapReduce applications. In:Proc. of the 2012 ACM SIGMOD Int'l Conf. on Management of Data. Arizona:ACM Press, 2012. 25-36.[doi:10.1145/2213836.2213840]
    [20] Cherniak A, Zaidi H, Zadorozhny V. Optimization strategies for A/B testing on HADOOP. Proc. of the VLDB Endowment, 2013, 6(11):973-984.[doi:10.14778/2536222.2536224]
    [21] Ekanayake J, Li H, Zhang B, Gunarathne T, Bae SH, Qiu J, Fox G. Twister:A runtime for iterative MapReduce. In:Proc. of the 19th ACM Int'l Symp. on High Performance Distributed Computing. Chicago:ACM Press, 2010. 810-818.[doi:10.1145/1851476. 1851593]
    [22] Bu Y, Howe B, Balazinska M, Ernst MD. HaLoop:Efficient iterative data processing on large clusters. Proc. of the VLDB Endowment, 2010,3(1-2):285-296.[doi:10.14778/1920841.1920881]
    [23] Zhang Y, Gao Q, Gao L, Wang C. iMapReduce:A distributed computing framework for iterative computation. Journal of Grid Computing, 2012,10(1):47-68.[doi:10.1007/s10723-012-9204-9]
    [24] Zaharia M, Chowdhury M, Das T, Dave A, Ma J, McCauley M, Franklin MJ, Shenker S, Stoica I. Resilient distributed datasets:A fault-tolerant abstraction for in-memory cluster computing. In:Proc. of the 9th USENIX Conf. on Networked Systems Design and Implementation. San Jose:USENIX Association Berkeley, 2012. 1-14.
    [25] Yu Y, Isard M, Fetterly D, Budiu M, Erlingsson Ú, Gunda PK, Currey J. DryadLINQ:A system for general-purpose distributed data-parallel computing using a high-level language. In:Proc. of the 8th USENIX Conf. on Operating Systems Design and Implementation. USENIX Association Berkeley, 2008. 1-14.
    [26] Lhoták O, Hendren L. Scaling Java points-to analysis using Spark. In:Proc. of the 12th Int'l Conf. on Compiler Construction. Berlin:Springer-Verlag, 2003. 153-169.
    [27] Ananthanarayanan G, Ghodsi A, Shenker S, Stoica I. Disk-Locality in datacenter computing considered irrelevant. In:Proc. of the 13th USENIX Conf. on Hot topics in Operating Systems. California USENIX Association Berkeley, 2011. 12-17.
    [28] Bhatotia P, Wieder A, Rodrigues R, Acar UA, Pasquin R. Incoop:MapReduce for incremental computations. In:Proc. of ACM Symp. on Cloud Computing. Cascais:ACM Press, 2011. 1-14.[doi:10.1145/2038916.2038923]
    [29] Armbrust M, Xin RS, Lian C, Huai Y, Liu D, Bradley JK, Meng X, Kaftan T, Franklin MJ, Ghodsi A. Spark sql:Relational data processing in Spark. In:Proc. of the 2015 ACM SIGMOD Int'l Conf. on Management of Data. Victoria:ACM Press, 2015. 1383-1394.[doi:10.1145/2723372.2742797]
    [30] Zaharia M, Das T, Li H, Hunter T, Shenker S, Stoica I. Discretized streams:Fault-tolerant streaming computation at scale. In:Proc. of the 24th ACM Symp. on Operating Systems Principles. Farmington:ACM Press, 2013. 423-438.[doi:10.1145/2517349. 2522737]
    [31] Meng X, Bradley J, Yavuz B, Sparks E, Venkataraman S, Liu D, Freeman J, Tsai D, Amde M, Owen S, Xin D, Reynold X, Franklin MJ, Zadeh R, Zaharia M, Talwalkar A. Mllib:Machine learning in apache Spark. The Journal of Machine Learning Research, 2016, 17(1):1235-1241.
    [32] Xin RS, Gonzalez JE, Franklin MJ, Stoica I. Graphx:A resilient distributed graph system on Spark. In:Proc. of the 1st Int'l Workshop on Graph Data Management Experiences and Systems. New York:ACM Press, 2013. 1-6.[doi:10.1145/2484425. 2484427]
    [33] Gonzalez JE, Xin RS, Dave A, Crankshaw D, Franklin MJ, Stoica I. GraphX:Graph processing in a distributed dataflow framework. In:Proc. of the 11th USENIX Conf. on Operating Systems Design and Implementation. USENIX Association Berkeley, 2014. 599-613.
    [34] Malewicz G, Austern MH, Bik AJ, Dehnert JC, Horn I, Leiser N, Czajkowski G. Pregel:A system for large-scale graph processing. In:Proc. of the 2010 ACM SIGMOD Int'l Conf. on Management of Data. ACM Press, 2010. 135-146.[doi:10.1145/1807167. 1807184]
    [35] Low Y, Bickson D, Gonzalez J, Guestrin C, Kyrola A, Hellerstein JM. Distributed GraphLab:A framework for machine learning and data mining in the cloud. Proc. of the VLDB Endowment, 2012,5(8):716-727.[doi:10.14778/2212351.2212354]
    [36] Thomas K, Grier C, Ma J, Paxson V, Song D. Design and evaluation of a real-time URL spam filtering service. In:Proc. of the 2011 IEEE Symp. on Security and Privacy (SP). Berkeley:IEEE, 2011. 447-462.[doi:10.1109/SP.2011.25]
    [37] Zhang Z, Barbary K, Nothaft FA, Sparks E, Zahn O, Franklin MJ, Patterson DA, Perlmutter S. Scientific computing meets big data technology:An astronomy use case. In:Proc. of the 2015 IEEE Int'l Conf. on Big Data. Santa Clara:IEEE, 2015. 918-927.[doi:10.1109/BigData.2015.7363840]
    [38] Nothaft FA, Massie M, Danford T, Zhang Z, Laserson U, Yeksigian C, Kottalam J, Ahuja A, Hammerbacher J, Linderman M. Rethinking data-intensive science using scalable analytics systems. In:Proc. of the 2015 ACM SIGMOD Int'l Conf. on Management of Data. Victoria:ACM Press, 2015. 631-646.[doi:10.1145/2723372.2742787]
    [39] Veiga J, Expósito RR, Pardo XC, Taboada GL, Tourifio J. Performance evaluation of big data frameworks for large-scale data analytics. In:Proc. of the 2016 IEEE Int'l Conf. on Big Data. Washington:IEEE, 2016. 424-431.[doi:10.1109/BigData. 2016. 7840633]
    [40] Lee H, Kang M, Youn SB, Lee JG, Kwon Y. An experimental comparison of iterative MapReduce frameworks. In:Proc. of the 25th ACM Int'l on Conf. on Information and Knowledge Management. Indiana:ACM Press, 2016. 2089-2094.[doi:10.1145/2983323.2983647]
    [41] Gu L, Li H. Memory or time:Performance evaluation for iterative operation on hadoop and Spark. In:Proc. of the 2013 IEEE 10th Int'l Conf. on High Performance Computing and Communications & 2013 IEEE Int'l Conf. on Embedded and Ubiquitous Computing (HPCC_EUC). Zhangjiajie:IEEE, 2013. 721-727.[doi:10.1109/HPCC.and.EUC.2013.106]
    [42] Kang M, Lee JG. An experimental analysis of limitations of MapReduce for iterative algorithms on Spark. Cluster Computing, 2017,20(4):3593-3604.[doi:10.1007/s10586-017-1167-y]
    [43] Mavridis I, Karatza H. Performance evaluation of cloud-based log file analysis with apache hadoop and apache Spark. Journal of Systems and Software, 2017,125:133-151.[doi:10.1016/j.jss.2016.11.037]
    [44] Song J, Sun ZZ, Mao KM, Bao YB, Yu G. Research advance on MapReduce based big data processing platforms and algorithms. Ruan Jian Xue Bao/Journal of Software, 2017,28(3):514-543(in Chinese with English abstract). http://www.jos.org.cn/1000-9825/5169.htm
    [45] Shi J, Qiu Y, Minhas UF, Jiao L, Wang C, Reinwald B, Özcan F. Clash of the titans:Mapreduce vs. Spark for large scale data analytics. Proc. of the VLDB Endowment, 2015,8(13):2110-2121.[doi:10.14778/2831360.2831365]
    [46] Page L. The PageRank citation ranking:Bringing order to the Web. Stanford Digital Libraries Working Paper, 1998,9(1):1-17.
    [47] Davidson A, Or A. Optimizing shuffle performance in Spark. Technical Report, University of California, Berkeley-Department of Electrical Engineering and Computer Sciences, 2013.
    [48] Wolf J, Balmin A, Rajan D, Hildrum K, Khandekar R, Parekh S, Wu KL, Vernica R. On the optimization of schedules for MapReduce workloads in the presence of shared scans. The Int'l Journal on Very Large Data Bases, 2012,21(5):589-609.[doi:10.1007/s00778-012-0279-5]
    附中文参考文献:
    [44] 宋杰,孙宗哲,毛克明,鲍玉斌,于戈.MapReduce大数据处理平台与算法研究进展.软件学报,2017,28(3):514-543. http://www.jos.org.cn/1000-9825/5169.htm[doi:10.13328/j.cnki.jos.005169]
    相似文献
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

吴信东,嵇圣硙. MapReduce与Spark用于大数据分析之比较.软件学报,2018,29(6):1770-1791

复制
分享
文章指标
  • 点击次数:
  • 下载次数:
  • HTML阅读次数:
  • 引用次数:
历史
  • 收稿日期:2017-10-19
  • 在线发布日期: 2018-02-08
文章二维码
友情链接:中国科学院软件研究所中国计算机学会中国科学院国家自然科学基金委员会CCF数字图书馆Int'l J of Softw Info计算机系统应用
您是第19876377位访问者
版权所有：中国科学院软件研究所 京ICP备05046678号-3
地址：北京市海淀区中关村南四街4号,邮政编码：100190
电话：010-62562563 传真：010-62562533 Email：jos@iscas.ac.cn
技术支持：北京勤云科技发展有限公司

京公网安备 11040202500063号