微信服务号

微信订阅号

2025-4-21- 13
Home > Archive>Volume 23, Issue 2, 2012 >253-265. DOI:10.3724/SP.J.1001.2012.04140
PDF HTML XML Export Cite reminder
Distributed Search Engine System Productivity Modeling and Evaluation
Author:
Affiliation:

  • Article
    • | |
  • Metrics
    • |
  • Reference [34]
    • |
  • Related [20]
    • |
  • Cited by [1]
    • | |
  • Comments
    Abstract:

    This study extends the current productivity models for a typical Web search engine system, which consists of a Web crawling system and an indexing system. Five different design schemata are characterized according to this model and are compared through power consumption, networking cost, system scale, and query efficiency. The half-WAN scheme, which consists of a WAN-based crawling system and a multi-cluster indexing system, is proved to be the best choice for a large-scale highly-efficient Web search engine.

    Reference
    [1] Baeza-Yates R, Gionis A, Junqueira F, Plachouras V, Telloli L. On the feasibility of multi-site Web search engines. In: Proc. of the 18th ACM Conf. on Information and Knowledge Management (CIKM 2009). ACM Press, 2009. 425-434. [doi: 10.1145/1645953.1646009]
    [2] Baeza-Yates R, Castillo C, Junqueira F, Plachouras V, Silvestri F. Challenges in distributed information retrieval. In: Proc. of the Int’l Conf. on Data Engineering (ICDE). Istanbul: IEEE Computer Society Press, 2007. 6-20. [doi: 10.1109/ICDE.2007.367846]
    [3] Church K, Greenberg A, Hamilton J. On delivering embarrassingly distributed cloud services. In: Proc. of the 7th ACM Workshop on Hot Topics in Networks Hotnets. New York: ACM Press, 2008. 1-6.
    [4] Cardwell N, Savage S, Anderson T. Modeling TCP latency. In: Proc. of the 19th Annual IEEE Int’l Conf. on Computer Communications 2000. IEEE Press, 2000. 1742-1751. [doi: 10.1109/INFCOM.2000.832574]
    [5] Cho J, Garcia-Molina H. Parallel crawlers. In: Proc. of the 11th Int’l Conf. on World Wide Web. 2002. 124-135. [doi: 10.1145/511446.511464]
    [6] Boldi P, Codenotti B, Santini M, Vigna S. Ubicrawler: A scalable fully distributed Web crawler. Software-Practice and Experience, 2004,34(8):711-726. [doi: 10.1002/spe.587]
    [7] Dustin B. Distributed high-performance Web crawlers: A survey of the state of the art. 2003. http://www.cs.ucsd.edu/~dboswell/PastWork/WebCrawlingSurvey.pdf
    [8] Papapetrou O, Samaras G. Ipmicra: An IP-address based location aware distributed Web crawler. In: Proc. of the 5th Int’l Conf. on Internet Computing (IC 2004). Las Vegas, 2004. 694-699.
    [9] Papapetrou O, Samaras G. Ipmicra: Toward a distributed and adaptable location aware Web crawler. In: Proc. of the 8th East European Conf. (ADBIS 2004). 2004. http://www.sztaki.hu/conferences/ADBIS/12-Samaras.pdf
    [10] Cambazoglu BB, Karaca E, Kucukyilmaz T, Turk A, Aykanat C. Architecture of a grid-enabled Web search engine. Information Processing and Management, 2007,43(3):609-623. [doi: 10.1016/j.ipm.2006.10.011]
    [11] Hafri Y, Djeraba C. High performance crawling system. In: Proc. of the 6th ACM SIGMM Int’l Workshop on Multimedia Information Retrieval. ACM Press, 2004. 299-306. [doi: 10.1145/1026711.1026760]
    [12] Singh A, Srivatsa M, Liu L, Miller T. Apoidea: A decentralized peer-to-peer architecture for crawling the World Wide Web. In: Proc. of the SIGIR 2003 Workshop on Distributed Information Retrieval. 2004. 126-142. [doi: 10.1007/978-3-540-24610-7_10]
    [13] Li XM, Yan HF, Wang JM. Search Engine: Principle, Technology and System. Beijing: Science Press, 2005 (in Chinese).
    [14] Liu F, Ma FY, Ye YM, Li ML, Yu JD. IGLOOG: A distributed Web crawler based on grid service. In: Proc. of the APWeb 2005. 2005. 207-216. [doi: 10.1007/978-3-540-31849-1_21]
    [15] Melink S, Raghavan S, Yang B, Garcia-Molina H. Building a distributed full-text index for the Web. ACM Trans. on Information System, 2001,19(3):217-241. [doi: 10.1145/502115.502116]
    [16] Tomasic A, Garcia-Molina H. Query processing and inverted indices in shared-nothing text document information retrieval systems. The VLDB Journal, 1993,2(3):243-276.
    [17] Moffat A, Webber W, Zobel J, Baeza-Yates R. A pipelined architecture for distributed text query evaluation. Information Retrieval, 2007,10(3):205-231. [doi: 10.1007/s10791-006-9014-4]
    [18] Badue CS, Baeza-Yates R, Ribeiro-Neto B, Ziviani A, Ziviani N. Analyzing imbalance among homogeneous index servers in a Web search system. Information Process Management, 2007,43(3):592-608. [doi: 10.1016/j.ipm.2006.09.002]
    [19] Moffat A, Webber W, Zobel J. Load balancing for term-distributed parallel retrieval. In: Proc. of the 29th Annual Int’l ACM SIGIR Conf. on Research and Development in Information Retrieval. 2006. 348-355. [doi: 10.1145/1148170.1148232]
    [20] Li JY, Loo BT, Hellerstein JM, Kaashoek F, Karger DR, Morris R. On the feasibility of peer-to-peer Web indexing and search. Lecture Notes in Computer Science, 2003,2735:207-215. [doi: 10.1007/978-3-540-45172-3_19]
    [21] Zhang J, Suel T. Efficient query evaluation on large textual collections in a peer-to-peer environment. In: Proc. of the 5th IEEE Int’l Conf. on Peer-to-Peer Computing. 2005. 225-233. [doi: 10.1109/P2P.2005.7]
    [22] Fagni T, Perego R, Silvestri F, Orlando S. Boosting the performance of Web search engines: Caching and prefetching query results by expoiting historical usage data. ACM Trans. on Information System, 2006,24(1):51-78. [doi: 10.1145/1125857.1125859]
    [23] Saraiva PC, de Moura ES, Ziviani N, Meira W, Fonseca R, Riberio-Neto B. Ranking-Preserving two-level caching for scalable search engines. In: Proc. of the 24th Annual Int’l ACM SIGIR Conf. on Research and Development in Information Retrieval. 2001. 51-58. [doi: 10.1145/383952.383959]
    [24] Baeza-Yates R, Gionis A, Junqueira F, Murdock V, Plachouras V, Silvestri F. The impact of caching on search engines. In: Proc. of the 30th Annual Int’l ACM SIGIR Conf. on Research and Development in Information Retrieval. 2007. 183-190. [doi: 10.1145/1277741.1277775]
    [25] Craswell N, Crimmins F, Hawking D, Moffat A. Performance and cost tradeoffs in Web search. In: Proc. of the 15th Australasian Database Conf. (ADC). 2004. 161-169.
    [26] Majestic-12: Distributed Web search. http://www.majestic12.co.uk/
    [27] Loo BT, Cooper O, Krishnamurthy S. Distributed Web crawling over DHTs. Technical Report, CSD-4-1305, Berkeley: Technical Department of Electrical Engineering and Computer Sciences, University of California, 2004. http://repository.upenn.edu/cgi/viewcontent.cgi?article=1345&context=cis_papers
    [28] Mcbryan OA. Genvl and WWWW: Tools for taming the Web. In: Proc. of the 1st Int’l World Wide Web Conf. 1994. 79-90.
    [29] Pinkerton B. Finding what people want: Experiences with the webcrawler. In: Proc. of the 1st World Wide Web Conf. Geneva, 1994. http://thinkpink.com/bp/WebCrawler/WWW94.html
    [30] Pinkerton B. Webcrawler: Finding what people want [Ph.D. Thesis]. Washington: University of Washington, 2000.
    [31] Lee HT, Leonard D, Wang XM, Loguinov D. Irlbot: Scaling to 6 billion pages and beyond. In: Proc. of the Int’l World Wide Web Conf. ACM Press, 2008. 427-436. [doi: 10.1145/1541822.1541823]
    [32] Eichmann D. The rbse spider—Balancing effective search against Web load. In: Proc. of the 1st Int’l World Wide Web Conf. 1994. 113-120.
    [33] Anderson DP, Cobb J, Korpela E, Lebofsky M, Werthimer D. SETI@home: An experiment in public-resource computing. Communications of the ACM, 2002,45(11):56-61. [doi: 10.1145/581571.581573]
    [34] Francis P, Jamin S, Jin C, Jin YX, Raz D, Shavitt Y, Zhang L. IDMAPS: A global Internet host distance estimation service. IEEE/ ACM Trans. on Networking, 2001,9(5):525-540. [doi: 10.1109/90.958323]
    Comments
    Comments
    分享到微博
    Submit
Get Citation

张伟哲,张宏莉,许笑,何慧.分布式搜索引擎系统效能建模与评价.软件学报,2012,23(2):253-265

Copy
Share
Article Metrics
  • Abstract:7781
  • PDF: 9436
  • HTML: 0
  • Cited by: 0
History
  • Received:July 08,2011
  • Revised:September 06,2011
  • Online: February 07,2012
Links:Institute of Software, CASChina Computer FederationCASNSFC
You are the first2036636Visitors
Copyright: Institute of Software, Chinese Academy of Sciences Beijing ICP No. 05046678-4
Address：4# South Fourth Street, Zhong Guan Cun, Beijing 100190,Postal Code：100190
Phone：010-62562563 Fax：010-62562533 Email：jos@iscas.ac.cn
Technical Support：Beijing Qinyun Technology Development Co., Ltd.

Beijing Public Network Security No. 11040202500063