微信服务号

微信订阅号

2025年3月24日 16:41 星期一
首页 > 过刊浏览>2012年第23卷第2期 >253-265. DOI:10.3724/SP.J.1001.2012.04140
PDF HTML阅读 XML下载 导出引用 引用提醒
分布式搜索引擎系统效能建模与评价
作者:
基金项目:

国家自然科学基金(61173145); 国家重点基础研究发展计划(973)(G2011CB302605); 国家高技术研究发展计划(863)(2010AA012504, 2011AA010705)


Distributed Search Engine System Productivity Modeling and Evaluation
Author:
  • 摘要
    • | |
  • 访问统计
    • |
  • 参考文献 [34]
    • |
  • 相似文献 [20]
    • |
  • 引证文献
    • | |
  • 文章评论
    摘要:

    针对分布式搜索引擎系统效能建模与评估问题,通过对当前分布式搜索引擎系统的建模与分类,扩展了能耗与网络开销的成本模型;对5种构建搜索引擎系统的设计方案,从系统成本、系统规模和查询响应时间等角度进行了详尽的理论分析与评价.由此发现,由广域网分布式采集系统和多机群索引系统组成的半广域网搜索引擎系统相对于其他系统具有相对较高的效能,同时能够较好地兼顾用户的服务质量.

    Abstract:

    This study extends the current productivity models for a typical Web search engine system, which consists of a Web crawling system and an indexing system. Five different design schemata are characterized according to this model and are compared through power consumption, networking cost, system scale, and query efficiency. The half-WAN scheme, which consists of a WAN-based crawling system and a multi-cluster indexing system, is proved to be the best choice for a large-scale highly-efficient Web search engine.

    参考文献
    [1] Baeza-Yates R, Gionis A, Junqueira F, Plachouras V, Telloli L. On the feasibility of multi-site Web search engines. In: Proc. of the 18th ACM Conf. on Information and Knowledge Management (CIKM 2009). ACM Press, 2009. 425-434. [doi: 10.1145/1645953.1646009]
    [2] Baeza-Yates R, Castillo C, Junqueira F, Plachouras V, Silvestri F. Challenges in distributed information retrieval. In: Proc. of the Int’l Conf. on Data Engineering (ICDE). Istanbul: IEEE Computer Society Press, 2007. 6-20. [doi: 10.1109/ICDE.2007.367846]
    [3] Church K, Greenberg A, Hamilton J. On delivering embarrassingly distributed cloud services. In: Proc. of the 7th ACM Workshop on Hot Topics in Networks Hotnets. New York: ACM Press, 2008. 1-6.
    [4] Cardwell N, Savage S, Anderson T. Modeling TCP latency. In: Proc. of the 19th Annual IEEE Int’l Conf. on Computer Communications 2000. IEEE Press, 2000. 1742-1751. [doi: 10.1109/INFCOM.2000.832574]
    [5] Cho J, Garcia-Molina H. Parallel crawlers. In: Proc. of the 11th Int’l Conf. on World Wide Web. 2002. 124-135. [doi: 10.1145/511446.511464]
    [6] Boldi P, Codenotti B, Santini M, Vigna S. Ubicrawler: A scalable fully distributed Web crawler. Software-Practice and Experience, 2004,34(8):711-726. [doi: 10.1002/spe.587]
    [7] Dustin B. Distributed high-performance Web crawlers: A survey of the state of the art. 2003. http://www.cs.ucsd.edu/~dboswell/PastWork/WebCrawlingSurvey.pdf
    [8] Papapetrou O, Samaras G. Ipmicra: An IP-address based location aware distributed Web crawler. In: Proc. of the 5th Int’l Conf. on Internet Computing (IC 2004). Las Vegas, 2004. 694-699.
    [9] Papapetrou O, Samaras G. Ipmicra: Toward a distributed and adaptable location aware Web crawler. In: Proc. of the 8th East European Conf. (ADBIS 2004). 2004. http://www.sztaki.hu/conferences/ADBIS/12-Samaras.pdf
    [10] Cambazoglu BB, Karaca E, Kucukyilmaz T, Turk A, Aykanat C. Architecture of a grid-enabled Web search engine. Information Processing and Management, 2007,43(3):609-623. [doi: 10.1016/j.ipm.2006.10.011]
    [11] Hafri Y, Djeraba C. High performance crawling system. In: Proc. of the 6th ACM SIGMM Int’l Workshop on Multimedia Information Retrieval. ACM Press, 2004. 299-306. [doi: 10.1145/1026711.1026760]
    [12] Singh A, Srivatsa M, Liu L, Miller T. Apoidea: A decentralized peer-to-peer architecture for crawling the World Wide Web. In: Proc. of the SIGIR 2003 Workshop on Distributed Information Retrieval. 2004. 126-142. [doi: 10.1007/978-3-540-24610-7_10]
    [13] Li XM, Yan HF, Wang JM. Search Engine: Principle, Technology and System. Beijing: Science Press, 2005 (in Chinese).
    [14] Liu F, Ma FY, Ye YM, Li ML, Yu JD. IGLOOG: A distributed Web crawler based on grid service. In: Proc. of the APWeb 2005. 2005. 207-216. [doi: 10.1007/978-3-540-31849-1_21]
    [15] Melink S, Raghavan S, Yang B, Garcia-Molina H. Building a distributed full-text index for the Web. ACM Trans. on Information System, 2001,19(3):217-241. [doi: 10.1145/502115.502116]
    [16] Tomasic A, Garcia-Molina H. Query processing and inverted indices in shared-nothing text document information retrieval systems. The VLDB Journal, 1993,2(3):243-276.
    [17] Moffat A, Webber W, Zobel J, Baeza-Yates R. A pipelined architecture for distributed text query evaluation. Information Retrieval, 2007,10(3):205-231. [doi: 10.1007/s10791-006-9014-4]
    [18] Badue CS, Baeza-Yates R, Ribeiro-Neto B, Ziviani A, Ziviani N. Analyzing imbalance among homogeneous index servers in a Web search system. Information Process Management, 2007,43(3):592-608. [doi: 10.1016/j.ipm.2006.09.002]
    [19] Moffat A, Webber W, Zobel J. Load balancing for term-distributed parallel retrieval. In: Proc. of the 29th Annual Int’l ACM SIGIR Conf. on Research and Development in Information Retrieval. 2006. 348-355. [doi: 10.1145/1148170.1148232]
    [20] Li JY, Loo BT, Hellerstein JM, Kaashoek F, Karger DR, Morris R. On the feasibility of peer-to-peer Web indexing and search. Lecture Notes in Computer Science, 2003,2735:207-215. [doi: 10.1007/978-3-540-45172-3_19]
    [21] Zhang J, Suel T. Efficient query evaluation on large textual collections in a peer-to-peer environment. In: Proc. of the 5th IEEE Int’l Conf. on Peer-to-Peer Computing. 2005. 225-233. [doi: 10.1109/P2P.2005.7]
    [22] Fagni T, Perego R, Silvestri F, Orlando S. Boosting the performance of Web search engines: Caching and prefetching query results by expoiting historical usage data. ACM Trans. on Information System, 2006,24(1):51-78. [doi: 10.1145/1125857.1125859]
    [23] Saraiva PC, de Moura ES, Ziviani N, Meira W, Fonseca R, Riberio-Neto B. Ranking-Preserving two-level caching for scalable search engines. In: Proc. of the 24th Annual Int’l ACM SIGIR Conf. on Research and Development in Information Retrieval. 2001. 51-58. [doi: 10.1145/383952.383959]
    [24] Baeza-Yates R, Gionis A, Junqueira F, Murdock V, Plachouras V, Silvestri F. The impact of caching on search engines. In: Proc. of the 30th Annual Int’l ACM SIGIR Conf. on Research and Development in Information Retrieval. 2007. 183-190. [doi: 10.1145/1277741.1277775]
    [25] Craswell N, Crimmins F, Hawking D, Moffat A. Performance and cost tradeoffs in Web search. In: Proc. of the 15th Australasian Database Conf. (ADC). 2004. 161-169.
    [26] Majestic-12: Distributed Web search. http://www.majestic12.co.uk/
    [27] Loo BT, Cooper O, Krishnamurthy S. Distributed Web crawling over DHTs. Technical Report, CSD-4-1305, Berkeley: Technical Department of Electrical Engineering and Computer Sciences, University of California, 2004. http://repository.upenn.edu/cgi/viewcontent.cgi?article=1345&context=cis_papers
    [28] Mcbryan OA. Genvl and WWWW: Tools for taming the Web. In: Proc. of the 1st Int’l World Wide Web Conf. 1994. 79-90.
    [29] Pinkerton B. Finding what people want: Experiences with the webcrawler. In: Proc. of the 1st World Wide Web Conf. Geneva, 1994. http://thinkpink.com/bp/WebCrawler/WWW94.html
    [30] Pinkerton B. Webcrawler: Finding what people want [Ph.D. Thesis]. Washington: University of Washington, 2000.
    [31] Lee HT, Leonard D, Wang XM, Loguinov D. Irlbot: Scaling to 6 billion pages and beyond. In: Proc. of the Int’l World Wide Web Conf. ACM Press, 2008. 427-436. [doi: 10.1145/1541822.1541823]
    [32] Eichmann D. The rbse spider—Balancing effective search against Web load. In: Proc. of the 1st Int’l World Wide Web Conf. 1994. 113-120.
    [33] Anderson DP, Cobb J, Korpela E, Lebofsky M, Werthimer D. SETI@home: An experiment in public-resource computing. Communications of the ACM, 2002,45(11):56-61. [doi: 10.1145/581571.581573]
    [34] Francis P, Jamin S, Jin C, Jin YX, Raz D, Shavitt Y, Zhang L. IDMAPS: A global Internet host distance estimation service. IEEE/ ACM Trans. on Networking, 2001,9(5):525-540. [doi: 10.1109/90.958323]
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

张伟哲,张宏莉,许笑,何慧.分布式搜索引擎系统效能建模与评价.软件学报,2012,23(2):253-265

复制
分享
文章指标
  • 点击次数:7767
  • 下载次数: 9365
  • HTML阅读次数: 0
  • 引用次数: 0
历史
  • 收稿日期:2011-07-08
  • 最后修改日期:2011-09-06
  • 在线发布日期: 2012-02-07
文章二维码
友情链接:中国科学院软件研究所中国计算机学会中国科学院国家自然科学基金委员会CCF数字图书馆Int'l J of Softw Info计算机系统应用
您是第19728364位访问者
版权所有：中国科学院软件研究所 京ICP备05046678号-3
地址：北京市海淀区中关村南四街4号,邮政编码：100190
电话：010-62562563 传真：010-62562533 Email：jos@iscas.ac.cn
技术支持：北京勤云科技发展有限公司

京公网安备 11040202500063号