网页变化与增量搜集技术
作者:
基金项目:

Supported by the National Natural Science Foundation of China under Grant Nos.60573166,60435020(国家自然科学基金);the National Research Foundation for the Doctoral Program of Higher Education of China under Grant No.20030001076(国家教育部博士点基金)

  • 摘要
  • | |
  • 访问统计
  • |
  • 参考文献 [97]
  • |
  • 相似文献 [20]
  • |
  • 引证文献
  • | |
  • 文章评论
    摘要:

    互联网络中信息量的快速增长使得增量搜集技术成为网上信息获取的一种有效手段,它可以避免因重复搜集未曾变化的网页而带来的时间和资源上的浪费.网页变化规律的发现和利用是增量搜集技术的一个关键.它用来预测网页的下次变化时间甚至变化程度;在此基础上,增量搜集系统还需要考虑网页的变化频率、变化程度和重要性,选择一种最优的任务调度算法来决定不同网页的搜集频率和相对搜集次序.针对网页变化和增量搜集技术这一主题,对最近几年的研究成果作总结,并介绍最新的研究进展.首先论述对网页变化规律的建模、模型参数估计和估计效率等问题;然后介绍几个著名的增量搜集系统,着重分析它们的任务调度算法;最后,从理论上分析和总结增量搜集系统的最佳任务调度算法及其一个基于启发式策略的近似解,并预测其将来的研究趋势.该工作对增量搜集系统的设计和Web演化规律的研究具有参考意义.

    Abstract:

    With the massive and ever increasing pages in the Web, incremental crawling has become a promising method to achieve on-line information. Its main advantage is the resource economization, which comes from the avoidance of downloading unchanged pages. For the precision of change prediction, the evolution of Web is generally studied to find out how pages change. In sum, incremental crawlers often integrate change frequency, change extent, and document quality for each page to determine its relative order as well as its download frequency. In this paper, the researches on Web evolution and incremental crawling in recent years are summarized: First, the change of page is modeled as a Poisson process, and the solutions are given to estimate its parameters, especially the change frequency, and then experimental results are shown. Second, based on the change of pages, three public large-scale incremental crawling systems are introduced, with emphasis on their scheduling policies and strategies to enhance page qualities. Third, theoretical analysis and exploration are performed to find the optimal scheduling policy, three approaches from different points of views are utilized to achieve this object, and a heuristic approximate solution is supplied for the feasibility in practice. Finally, research trends in this area are predicted, and three main issues are listed.

    参考文献
    [1]Shkapenyuk V,Suel T.Design and implementation of a high-performance distributed Web crawler.In:Proc.of the 18th Int'l Conf.on Data Engineering.San Jose:IEEE Press,2002.357-368.
    [2]Brin S,Page L.The anatomy of a large-scale hypertextual Web search engine.Computer Networks,1998,30(1-7):107-117.
    [3]Burner M.Crawling towards eternity:Building an archive of the World Wide Web.Web Techniques Magazine,1997,2(5):37-40.
    [4]Boldi P,Codenotti B,Santini M,Vigna S.Ubicrawler:A scalable fully distributed Web crawler.Software:Practice & Experience,2004,34(8):711-726.
    [5]Yan HF,Wang JY,Li XM,Guo L.Architectural design and evaluation of an efficient Web-crawling system.Journal of Systems and Software,2002,60(3):185-193.
    [6]Najork M,Heydon A.High-Performance Web crawling.Research Report,173,Compaq Systems Research Center,2001.
    [7]Hafri Y,Djeraba C.High performance crawling system.In:Proc.of the 6th ACM SIGMM Int'l Workshop on Multimedia Information Retrieval.New York:ACM Press,2004.299-306.
    [8]Broder A,Kumar R,Maghoul F,Raghavan P,Rajagopalan S,Stata R,Tomkins A,Wiener J.Graph structure in the Web:Experiments and models.In:Proc.of the 9th WWW Conf.North-Holland:Elsevier Science Publishers,2000.309-320.
    [9]Kleinberg J,Kumar R,Raghavan P,Rajagopalan S,Tompkins A.The Web as a graph:Measurements,models,and methods.Lecture Notes in Computer Science,1999,1627:1-18.
    [10]Cooper C,Frieze A.Crawling on simple models of Web graphs.Internet Mathematics,2003,1(1):57-90.
    [11]Broder AZ,Najork M,Wiener JL.Efficient URL caching for World Wide Web crawling.In:Proc.of the 12th Int'l World Wide Web Conf.New York:ACM Press,2003.679-689.
    [12]Fetterly D,Manasse M,Najork M.Spam,damn Spam,and statistics:Using statistical analysis to locate Spam Web pages.In:Amer-Yahia S,Gravano L,eds.Proc.of the 7th Int'l Workshop on the Web and Databases (WebDB 2004).New York:ACM Press,2004.1-6.
    [13]Najork M,Wiener JL.Breadth-First crawling yields high-quality pages.In:Proc.of the 10th Int'l World Wide Web Conf.North-Holland:Elsevier Science Publishers,2001.114-118.
    [14]Bharat K,Broder A.Mirror,mirror on the Web:A study of host pairs with replicated content.In:Proc.of the 8th Int'l World-Wide Web Conf.North-Holland:Elsevier Science Publishers,1999.501-512.
    [15]Li XM,Feng WS.Two effective functions on hashing URL.Journal of Software,2004,15(2):179-184 (in Chinese with English abstract).http://www.j os.org.cn/1000-9825/15/179.htm
    [16]Cho J,Garcia-Molina H.Parallel crawlers.In:Proc.of the 11th World Wide Web Conf.New York:ACM Press,2002.124-135.
    [17]Cho J,Garcia-Molina H.The evolution of the Web and implications for an incremental crawler.In:Proc.of the 26th Int'l Conf.on Very Large Databases.San Francisco:Morgan Kaufmann Publishers,2000.200-209.
    [18]Douglis F,Feldmann A,Krishnamurthy B.Rate of change and other metrics:A live study of the World Wide Web.In:Proc.of the USENIX Symp.on Internet Technologies and Systems.1997.147-158.
    [19]Fetterly D,Manasse M,Najork M,Wiener J.A large-scale study of the evolution of Web pages.In:Proc.of the 12th Int'l World Wide Web Conf.New York:ACM Press,2003.669-678.
    [20]Fetterly D,Manasse M,Najork M.On the evolution of clusters of near-duplicate Web pages.In:Proc.of the 1st Latin American Web Congress.2003.37-45.
    [21]Brewington BE,Cybenko G.How dynamic is the Web? In:Proc.of the 9th Int'l World Wide Web Conf.North-Holland:Elsevier Science Publishers,2000.257-276.
    [22]Brewington BE,Cybenko G.Keeping up with the changing Web.Computer,2000,33(5):52-58.
    [23]Luis FR,Shipman F,Karadkar U,Furuta R,Arora A.Perception of content,structure,and presentation changes in Web-based hypertext.In:Proc.of the ACM Conf.on Hypertext.New York:ACM Press,2001.205-214.
    [24]Meng T,Yan HF,Wang JM,Li XM.The evolution of link-attributes for pages and its implications on Web crawling.In:Proc.of the 2004 IEEE/WIC/ACM Int'l Conf.on Web Intelligence.Washington:IEEE Computer Society Press,2004.578-581.
    [25]Cho J,Garcia-Molina H.Synchronizing a database to improve freshness.In:Proc.of the 2000 ACM Int'l Conf.on Management of Data.New York:ACM Press,2000.117-128.
    [26]Cho J,Ntoulas A.Effective change detection using sampling.In:Proc.of the 28th Int'l Conf.on Very Large Databases.San Francisco:Morgan Kaufmann Publishers,2002.514-525.
    [27]Ntoulas A,Cho J,Olston C.What's new on the Web? The evolution of the Web from a search engine perspective.In:Proc.of the 13th World-Wide Web Conf.New York:ACM Press,2004.1-12.
    [28]Ipeirotis PG,Ntoulas A,Cho J,Gravano L.Modeling and managing content changes in text databases.In:Proc.of the Int'l Conf.on Data Engineering.Washington:Computer Society Press of the IEEE,2005.606-617.
    [29]Cho J,Garcia-Molina H.Estimating frequency of change.ACM Trans.on Internet Technology,2003,3(3):256-290.
    [30]Padmanabhan VN,Qiu L.The content and access dynamics of a busy Web site:Findings and implications.In:Proc.of the ACM SIGCOMM Conf.New York:ACM,2000.111-123.
    [31]Li XM.An estimation of the quantity of Web pages ever in China.Journal of Peking University (Science and Technology),2003,39(3):394-398 (in Chinese with English abstract).
    [32]Bar-Ilan J,Peritz BD.Evolution,continuity,and disappearance of documents on a specific topic on the Web:A longitudinal study of "informetrics".Journal of the American Society for Information Science and Technology,2004,55(11):980-990.
    [33]Pandey S,Olston C.User-Centric Web crawling.In:Proc.of the 14th Int'l Conf.on World Wide Web.New York:ACM Press,2005.401-411.
    [34]Arasu A,Cho J,Garcia-Molina H,Paepcke A,Raghavan S.Searching the Web.ACM Trans.on Internet Technology,2001,1(1):2-43.
    [35]Ceaparu I,Shneiderman B.Finding governmental statistical data on the Web:A study of categorically organized links for the FedStats topics page.Journal of the American Society for Information Science and Technology,2004,55 (11):1008-1015.
    [36]Toyoda M,Kitsuregawa M.Extracting evolution of Web communities from a series of Web archives.In:Proc.of the 14th ACM Conf.on Hypertext and hypermedia.New York:ACM Press,2003.28-37.
    [37]Meng T,Yan HF,Wang JM.Characterizing temporal locality in changes of Web documents.Journal of the China Society for Scientific and Technical Information,2005,24(4):398-406 (in Chinese with English abstract).
    [38]Koehler W.Web page change and persistence-A four-year longitudinal study.Journal of the American Society for Information Science and Technology,2002,53(2):162-171.
    [39]Chiang WTM,Hagenbuchner M,Tsoi AC.The WT10G dataset and the evolution of the Web.In:Proc.of the 14th Int'l Conf.on World Wide Web.New York:ACM Press,2005.938-939.
    [40]Dalal Z,Dash S,Dave P,Francisco-Revilla L,Furuta R,Karadkar U,Shipman F.Managing distributed collections:evaluating Web page changes,movement,and replacement.In:Proc.of the 4th ACM/IEEE-CS Joint Conf.on Digital Libraries.New York:ACM Press,2004.160-168.
    [41]Cho J,Roy S.Impact of Web search engines on page popularity.In:Proc.of the 13th World-Wide Web Conf.New York:ACM Press,2004.20-29.
    [42]Baeza-Yates RA,Saint-Jean F,Castillo C.Web structure,dynamics and page quality.Lecture Notes in Computer Science,2002,2476:117-130.
    [43]Adamic LA,Huberman BA.Power-Law distribution of the World Wide Web.Science,2000,287:2115.
    [44]Cho J,Roy S,Adams RE.Page quality:In search of an unbiased Web ranking.In:Proc.of the 2005 ACM Int'l Conf.on Management of Data.New York:ACM Press,2005.
    [45]Page L,Brin S,Motwani R,Winograd T.The PageRank citation ranking:Bringing order to the Web.Technology Report,1998.http://www-db.stanford.edu/~backrub/pageranksub.ps
    [46]Kleinberg JM.Authoritative sources in a hyperlinked environment.Journal of the ACM,1999,46(5):604-632.
    [47]Raghavan VV,Wong SKM.A critical analysis of vector space model for information retrieval.Journal of the American Society for Information Science,1986,37(5):279-287.
    [48]Edwards J,McCurley K,Tomlin J.An adaptive model for optimizing performance of an incremental Web crawler.In:Proc.of the 10th Int'l Conf.on World Wide Web.New York:ACM Press,2001.106-113.
    [49]Castillo C,Baeza-Yates R.A new model for Web crawling.In:Proc.of the 11th World Wide Web Conf.New York:ACM Press,2002.
    [50]Heydon A,Najork M.Mercator:A scalable,extensible Web crawler.World Wide Web,1999,2(4):219-229.
    [51]Meng T,Yan HF,Wang JM.A model of efficient incremental spider for the Chinese Web and its implementation.Journal of Tsinghua University (Science and Technology),2005,45(S 1):1882-1886 (in Chinese with English abstract).
    [52]Cho J.Crawling the Web:Discovery and maintenance of large-scale Web data[Ph.D.Thesis].Stanford:Stanford University,2001.
    [53]Baeza-Yates R,Castillo C.Crawling the infinite Web:five levels are enough.In:Proc.of the 3rd Workshop on Web Graphs.Berlin:Springer-Verlag,2004.156-167.
    [54]Cho J,Garcia-Molina H,Page L.Efficient crawling through URL ordering.Computer Networks,1998,30(1-7):161-172.
    [55]Baeza-Yates R,Castillo C,Marin M,Rodriguez A.Crawling a country:Better strategies than breadth-first for Web page ordering.In:Proc.of the 14th Int'l Conf.on World Wide Web.New York:ACM Press,2005.864-872.
    [56]Abiteboul S,Preda M,Cobena G.Adaptive on-line page importance computation.In:Proc.of the 12th Int'l Conf.on World Wide Web.New York:ACM Press,2003.280-290.
    [57]Davison BD.Topical locality in the Web.In:Proc.of the 23rd Annual Int'l ACM SIGIR Conf.on Research and Development in Information Retrieval.New York:ACM Press,2000.272-279.
    [58]Henzinger M,Motwani R,Silverstein C.Challenges in Web search engines.SIGIR Forum,2002,36(2):1-8.
    [59]Cope J,Craswell N,Hawking D.Automatic discovery of search interfaces on the Web.In:Proc.of the 14th Australasian Database Conf.2003.181-189.
    [60]Bharat K,Broder A,Henzinger MR.A comparison of techniques to find mirrored hosts on the WWW.Journal of the American Society and Information Science,2000,5 l(12):1114-1122.
    [61]Sivasubramanian S,Szymaniak M,Pierre G,van Steen M.Replication for Web hosting systems.ACM Computing Surveys,2004,36(3):291-334.
    [62]Perkins A.White paper:The classification of search engine spam.2001.http://www.silverdisc.co.uk/articles/spam-classification/
    [63]Davison B.Recognizing nepotistic links on the Web.In:Proc.of the AAAI 2000 Workshop on Artificial Intelligence for Web Search.San Jose:AAAI Press,2000.23-28.
    [64]Gyongyi Z,Garcia-Molina H,Pedersen J.Combatting Web spam with Trustrank.In:Proc.of the 30th VLDB Conf.San Francisco:Morgan Kaufmann Publishers,2004.576-587.
    [65]Wu BN,Davison BD.Identifying link farm spam pages.In:Proc.of the 14th Int'l Conf.on World Wide Web.New York:ACM Press,2005.820-829.
    [66]Fetterly D,Manasse M,Najork M.Detecting phrase-level duplication on the World Wide Web.In:Proc.of the SIGIR 2005.New York:ACM Press,2005.170-177.
    [67]Czumaj A,Finch I,Gasieniec L,Gibbons A,Leng PH,Rytter W,Zito M.Efficient Web searching using temporal factors.Theoretical Computer Science,2001,262(1-2):569-582.
    [68]Koster M.Robots in the Web:Threat or treat? ConneXions,1995,9(4):2-12.
    [69]Craswell N,Crimmins F,Hawking D,Moffat A.Performance and cost tradeoffs in Web search.In:Proc.of the 15th Australasian Conf.on Database.2004.161-169.
    [70]Talim J,Liu Z,Nain P,Coffman E.Optimizing the number of robots for Web search engines.Telecommunication Systems Journal,2001,17(1-2):234-243.
    [71]Broder A,Glassman S,Manasse M,Zweig G.Syntactic clustering of the Web.In:Proc.of the 6th Int'l World Wide Web Conf.New York:ACM,1997.1157-1166.
    [72]Dyreson CE,Lin HL,Wang YX.Managing versions of Web documents in a transaction-time Web server.In:Proc.of the 13th Int'l Conf.on World Wide Web.New York:ACM Press,2004.422-432.
    [73]Carney D,Lee S,Zdonik S.Scalable application-aware data freshening.In:Proc.of the 18th Int'l Conf.on Data Engineering.California:ACM Press,2003.481-492.
    [74]Hah JC,Cercone N,Hu XH.A weighted freshness metric for maintaining search engine local repository.In:Proc.of the 2004 IEEE/WIC/ACM Int'l Conf.on Web Intelligence.Washington:IEEE Press,2004.677-680.
    [75]Wolf JL,Squillante MS,Yu PS,Sethuraman J,Ozsen L.Optimal crawling strategies for Web search engines.In:Proc.of the 11th Int'l World Wide Web Conf.New York:ACM Press,2002.136-147.
    [76]Bullot H,Gupta SK,Mohania MK.A data-mining approach for optimizing performance of an incremental crawler.In:Proc.of the IEEE/WIC Int'l Conf.on Web Intelligence.Washington:IEEE Press,2003.610-613.
    [77]Cho J,Garcia-Molina H.Effective page refresh policies for Web crawlers.ACM Trans.on Database Systems,2003,28(4):390-426.
    [78]Jr Coffman EG,Liu Z,Weber RR.Optimal robot scheduling for Web search engines.Journal of Scheduling,1998,1(1):15-29.
    [79]Sigman K.Stationary Marked Point Process:An Intuitive Approach.New York:Chapman and Hall,1995.
    [80]Taylor HM,Karlin S.An Introduction To Stochastic Modeling.3rd ed.,New York:Academic Press,1998.
    [81]Borst S,Boxma OJ,Harink JHA,Huitema GB.Optimization of Fixed time polling schemes.Telecommunications Systems,1994,3(1):31-59.
    [82]Boxma OJ,Levy H,Weststrate JA.Efficient visit orders for polling systems.Performance Evaluation,1993,18(2):103-123.
    [83]Ibaraki T,Katoh N.Resource Allocation Problems:Algorithmic Approaches.Cambridge:MIT Press,1988.
    [84]Chakrabarti S.Mining the Web:Discovering Knowledge from Hypertext Data.San Francisco:Morgan-Kauffman Publishers,2002.
    [85]Xu BW,Zhang WF.Search Engines and Information Retrieval Technology.Beijing:Tsinghua University Press,2003 (in Chinese).
    [86]Li XM,Yan HF,Wang JM.Search Engine-Principle,Technology and System.Beijing:Science Press,2005 (in Chinese).
    [87]Meng T,Yan HF,Li XM.An evaluation model on information coverage of search engines.Chinese Journal of Electronics,2003,31(8):1168-1172 (in Chinese with English abstract).
    [88]Naumann F,Freytag JC,Leser U.Completeness of integrated information sources.Information Systems,2004,29(7):583-615.
    [89]Vaughan L,Thelwall M.Search engine coverage bias:Evidence and possible causes.Information Processing and Management:An Int'l Journal,2004,40(4):693-707.
    [90]Boldi P,Vigna S.The Webgraph framework I:Compression techniques.In:Proc.of the 13th World Wide Web Conf.New York:ACM Press,2004.595-601.
    [15]李晓明,凤旺森.两种对URL的散列效果很好的函数.软件学报,2004,15(2):179-184.http://www.jos.org.cn/1000-9825/15/179.htm
    [31]李晓明.对中国曾有过静态网页数的一种估计.北京大学学报(自然科学版),2003,39(3):394-398.
    [37]孟涛,闫宏飞,王继民.Web网页信息变化的时间局部性规律及其验证.情报学报,2005,24(4):398-406.
    [51]孟涛,闫宏飞,王继民.一个增量搜集中国Web的系统模型及其实现.清华大学学报(自然科学版),2005,45(S1):1882-1886.
    [85]徐宝文,张卫丰.搜索引擎与信息获取技术.北京:清华大学出版社,2003.
    [86]李晓明,闫宏飞,王继民.搜索引擎--原理、技术与系统.北京:科学出版社,2005.
    [87]孟涛,闫宏飞,李晓明.一个评价搜索引擎信息覆盖率的模型及其验证.电子学报,2003,31(8):1168-1172.
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

孟涛,王继民,闫宏飞.网页变化与增量搜集技术.软件学报,2006,17(5):1051-1067

复制
分享
文章指标
  • 点击次数:8832
  • 下载次数: 8591
  • HTML阅读次数: 0
  • 引用次数: 0
历史
  • 收稿日期:2005-10-11
  • 最后修改日期:2006-01-12
文章二维码
您是第19867622位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62562563 传真:010-62562533 Email:jos@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号