Web crawler (English Wikipedia)

Analysis of information sources in references of the Wikipedia article "Web crawler" in English language version.

Last modified: 16:00:26, 20 September 2026 (UTC)

Ref.Un. Ref.Website

Global rank English rank

2724doi.org

2^nd place

2626web.archive.org

1^st place

1311semanticscholar.org

15^th place

8^th place

33chato.cl

low place

33researchgate.net

96^th place

89^th place

22www10.org

low place

22adsabs.harvard.edu

14^th place

9^th place

22unimi.it

9,149^th place

9,365^th place

22wlv.ac.uk

low place

22indiana.edu

2,374^th place

1,453^rd place

22robotstxt.org

low place

22stanford.edu

194^th place

182^nd place

11webbrowsersintroduction.com

low place

11developers.google.com

3,679^th place

7,859^th place

11uiuc.edu

4,225^th place

3,432^nd place

11foaf-project.org

low place

11springer.com

182^nd place

172^nd place

11nih.gov

5^th place

11ucla.edu

885^th place

584^th place

11www2003.org

low place

11uiowa.edu

2,935^th place

1,725^th place

11fxpal.com

low place

11psu.edu

299^th place

159^th place

11hdl.handle.net

66^th place

34^th place

11brown.edu

2,455^th place

1,451^st place

11nyu.edu

1,362^nd place

822^nd place

11csic.es

2,036^th place

2,704^th place

11mccurley.org

low place

11poly.edu

low place

11doria.fi

5,349^th place

low place

11arxiv.org

49^th place

31^st place

11mendeley.com

low place

11support.google.com

2,836^th place

2,952^nd place

11semanticweb.com

low place

11apple.com

70^th place

55^th place

11wired.com

254^th place

178^th place

11canada.ca

1,114^th place

605^th place

adsabs.harvard.edu (Global: 14^th place; English: 9^th place)

ui.adsabs.harvard.edu

Lawrence, Steve; C. Lee Giles (8 July 1999). "Accessibility of information on the web". Nature. 400 (6740): 107–9. Bibcode:1999Natur.400..107L. doi:10.1038/21987. PMID 10428673. S2CID 4347646.
Michael L Nelson; Herbert Van de Sompel; Xiaoming Liu; Terry L Harrison; Nathan McFarland (24 March 2005). "mod_oai: An Apache Module for Metadata Harvesting": cs/0503069. arXiv:cs/0503069. Bibcode:2005cs........3069N. {{cite journal}}: Cite journal requires |journal= (help)

apple.com (Global: 70^th place; English: 55^th place)

support.apple.com

"About Applebot". Apple Inc. Retrieved 18 October 2021.

arxiv.org (Global: 49^th place; English: 31^st place)

Michael L Nelson; Herbert Van de Sompel; Xiaoming Liu; Terry L Harrison; Nathan McFarland (24 March 2005). "mod_oai: An Apache Module for Metadata Harvesting": cs/0503069. arXiv:cs/0503069. Bibcode:2005cs........3069N. {{cite journal}}: Cite journal requires |journal= (help)

brown.edu (Global: 2,455^th place; English: 1,451^st place)

cs.brown.edu

Junghoo Cho; Hector Garcia-Molina (2000). "Synchronizing a database to improve freshness" (PDF). Proceedings of the 2000 ACM SIGMOD international conference on Management of data. Dallas, Texas, United States: ACM. pp. 117–128. doi:10.1145/342009.335391. ISBN 1-58113-217-4. Retrieved 23 March 2009.

canada.ca (Global: 1,114^th place; English: 605^th place)

"Xenon web crawling initiative: privacy impact assessment (PIA) summary". Ottawa: Government of Canada. 11 April 2017. Archived from the original on 25 September 2017. Retrieved 13 October 2017.

chato.cl (Global: low place; English: low place)

Castillo, Carlos (2004). Effective Web Crawling (PhD thesis). University of Chile. Retrieved 3 August 2010.
Baeza-Yates, R.; Castillo, C.; Marin, M. and Rodriguez, A. (2005). "Crawling a Country: Better Strategies than Breadth-First for Web Page Ordering." In: Proceedings of the Industrial and Practical Experience track of the 14th conference on World Wide Web, pages 864–872, Chiba, Japan. ACM Press.
Baeza-Yates, R. and Castillo, C. (2002). Balancing volume, quality and freshness in Web crawling. In Soft Computing Systems – Design, Management and Applications, pages 565–572, Santiago, Chile. IOS Press Amsterdam.

csic.es (Global: 2,036^th place; English: 2,704^th place)

cindoc.csic.es

Heydon, Allan; Najork, Marc (26 June 1999). "Mercator: A Scalable, Extensible Web Crawler" (PDF). Archived from the original (PDF) on 19 February 2006. Retrieved 22 March 2009. {{cite journal}}: Cite journal requires |journal= (help)

developers.google.com (Global: 3,679^th place; English: 7,859^th place)

"In-depth guide to how Google Search works". Google Search Central. Google. Retrieved 11 June 2026.

doi.org (Global: 2^nd place; English: 2^nd place)

Lawrence, Steve; Giles, C. Lee (1998). "Searching the World Wide Web". Science. 280 (5360): 98–100. doi:10.1126/science.280.5360.98.
Lawrence, Steve; Giles, C. Lee (1999). "Accessibility of information on the web". Nature. 400 (6740): 107–109. doi:10.1038/21987.
Kobayashi, M. & Takeda, K. (2000). "Information retrieval on the web". ACM Computing Surveys. 32 (2): 144–173. doi:10.1145/358923.358934. S2CID 3710903.
Edwards, J.; McCurley, K. S.; and Tomlin, J. A. (2001). "An adaptive model for optimizing performance of an incremental web crawler". Proceedings of the 10th international conference on World Wide Web. pp. 106–113. doi:10.1145/371920.371960. ISBN 978-1581133486. S2CID 10316730. Archived from the original on 25 June 2014. Retrieved 25 January 2007.{{cite book}}: CS1 maint: multiple names: authors list (link)
Gulls, A.; A. Signori (2005). "The indexable web is more than 11.5 billion pages". Special interest tracks and posters of the 14th international conference on World Wide Web. ACM Press. pp. 902–903. doi:10.1145/1062745.1062789.
Lawrence, Steve; C. Lee Giles (8 July 1999). "Accessibility of information on the web". Nature. 400 (6740): 107–9. Bibcode:1999Natur.400..107L. doi:10.1038/21987. PMID 10428673. S2CID 4347646.
Cho, J.; Garcia-Molina, H.; Page, L. (April 1998). "Calorimetry in High Energy Physics". Seventh International World-Wide Web Conference. Brisbane, Australia. doi:10.1142/3725. ISBN 978-981-02-3400-3. Retrieved 23 March 2009.{{cite journal}}: CS1 maint: periodical has ISBN (link)
Abiteboul, Serge; Mihai Preda; Gregory Cobena (2003). "Adaptive on-line page importance computation". Proceedings of the 12th international conference on World Wide Web. Budapest, Hungary: ACM. pp. 280–290. doi:10.1145/775152.775192. ISBN 1-58113-680-3. Archived from the original on 20 March 2008. Retrieved 22 March 2009.
Boldi, Paolo; Bruno Codenotti; Massimo Santini; Sebastiano Vigna (2004). "UbiCrawler: a scalable fully distributed Web crawler" (PDF). Software: Practice and Experience. 34 (8): 711–726. doi:10.1002/spe.587. S2CID 325714. Archived from the original (PDF) on 20 March 2009. Retrieved 23 March 2009.
Boldi, Paolo; Massimo Santini; Sebastiano Vigna (2004). "Do Your Worst to Make the Best: Paradoxical Effects in PageRank Incremental Computations" (PDF). Algorithms and Models for the Web-Graph. Lecture Notes in Computer Science. Vol. 3243. pp. 168–180. doi:10.1007/978-3-540-30216-2_14. ISBN 978-3-540-23427-2. Archived from the original (PDF) on 1 October 2005. Retrieved 23 March 2009.
Cothey, Viv (2004). "Web-crawling reliability" (PDF). Journal of the American Society for Information Science and Technology. 55 (14): 1228–1238. doi:10.1002/asi.20078. Archived from the original (PDF) on 22 September 2017. Retrieved 26 October 2017.
Chakrabarti, Soumen; Van Den Berg, Martin; Dom, Byron (1999). "Focused crawling: A new approach to topic-specific Web resource discovery" (PDF). Computer Networks. 31 (11–16): 1623–1640. doi:10.1016/s1389-1286(99)00052-3. Archived from the original (PDF) on 17 March 2004.
Wu, Jian; Teregowda, Pradeep; Khabsa, Madian; Carman, Stephen; Jordan, Douglas; San Pedro Wandelmer, Jose; Lu, Xin; Mitra, Prasenjit; Giles, C. Lee (2012). "Web crawler middleware for search engine digital libraries". Proceedings of the twelfth international workshop on Web information and data management - WIDM '12. p. 57. doi:10.1145/2389936.2389949. ISBN 9781450317207. S2CID 18513666.
Wu, Jian; Teregowda, Pradeep; Ramírez, Juan Pablo Fernández; Mitra, Prasenjit; Zheng, Shuyi; Giles, C. Lee (2012). "The evolution of a crawling strategy for an academic document search engine". Proceedings of the 3rd Annual ACM Web Science Conference on - Web Sci '12. pp. 340–343. doi:10.1145/2380718.2380762. ISBN 9781450312288. S2CID 16718130.
Dong, Hai; Hussain, Farookh Khadeer; Chang, Elizabeth (2009). "State of the Art in Semantic Focused Crawlers". Computational Science and Its Applications – ICCSA 2009. Lecture Notes in Computer Science. Vol. 5593. pp. 910–924. doi:10.1007/978-3-642-02457-3_74. hdl:20.500.11937/48288. ISBN 978-3-642-02456-6.
Dong, Hai; Hussain, Farookh Khadeer (2013). "SOF: A semi-supervised ontology-learning-based focused crawler". Concurrency and Computation: Practice and Experience. 25 (12): 1755–1770. doi:10.1002/cpe.2980. S2CID 205690364.
Junghoo Cho; Hector Garcia-Molina (2000). "Synchronizing a database to improve freshness" (PDF). Proceedings of the 2000 ACM SIGMOD international conference on Management of data. Dallas, Texas, United States: ACM. pp. 117–128. doi:10.1145/342009.335391. ISBN 1-58113-217-4. Retrieved 23 March 2009.
E. G. Coffman Jr; Zhen Liu; Richard R. Weber (1998). "Optimal robot scheduling for Web search engines". Journal of Scheduling. 1 (1): 15–29. doi:10.1002/(SICI)1099-1425(199806)1:1<15::AID-JOS3>3.0.CO;2-K.
Cho, Junghoo; Garcia-Molina, Hector (2003). "Effective page refresh policies for Web crawlers". ACM Transactions on Database Systems. 28 (4): 390–426. doi:10.1145/958942.958945. S2CID 147958.
Junghoo Cho; Hector Garcia-Molina (2003). "Estimating frequency of change". ACM Transactions on Internet Technology. 3 (3): 256–290. doi:10.1145/857166.857170. S2CID 9362566.
Dill, S.; Kumar, R.; Mccurley, K. S.; Rajagopalan, S.; Sivakumar, D.; Tomkins, A. (2002). "Self-similarity in the web" (PDF). ACM Transactions on Internet Technology. 2 (3): 205–223. doi:10.1145/572326.572328. S2CID 6416041.
M. Thelwall; D. Stuart (2006). "Web crawling ethics revisited: Cost, privacy and denial of service". Journal of the American Society for Information Science and Technology. 57 (13): 1771–1779. doi:10.1002/asi.20388. Archived from the original on 7 March 2008. Retrieved 28 January 2008.
Brin, Sergey; Page, Lawrence (1998). "The anatomy of a large-scale hypertextual Web search engine". Computer Networks and ISDN Systems. 30 (1–7): 107–117. doi:10.1016/s0169-7552(98)00110-x. S2CID 7587743.
Shestakov, Denis; Bhowmick, Sourav S.; Lim, Ee-Peng (2005). "DEQUE: Querying the Deep Web" (PDF). Data & Knowledge Engineering. 52 (3): 273–311. doi:10.1016/s0169-023x(04)00107-7.^{[permanent dead link]}

doria.fi (Global: 5,349^th place; English: low place)

oa.doria.fi

Shestakov, Denis (2008). Search Interfaces on the Web: Querying and Characterizing Archived 6 July 2014 at the Wayback Machine. TUCS Doctoral Dissertations 104, University of Turku

foaf-project.org (Global: low place; English: low place)

wiki.foaf-project.org

See definition of scutter on FOAF Project's wiki Archived 13 December 2009 at the Wayback Machine

fxpal.com (Global: low place; English: low place)

Chakrabarti, Soumen; Van Den Berg, Martin; Dom, Byron (1999). "Focused crawling: A new approach to topic-specific Web resource discovery" (PDF). Computer Networks. 31 (11–16): 1623–1640. doi:10.1016/s1389-1286(99)00052-3. Archived from the original (PDF) on 17 March 2004.

hdl.handle.net (Global: 66^th place; English: 34^th place)

Dong, Hai; Hussain, Farookh Khadeer; Chang, Elizabeth (2009). "State of the Art in Semantic Focused Crawlers". Computational Science and Its Applications – ICCSA 2009. Lecture Notes in Computer Science. Vol. 5593. pp. 910–924. doi:10.1007/978-3-642-02457-3_74. hdl:20.500.11937/48288. ISBN 978-3-642-02456-6.

indiana.edu (Global: 2,374^th place; English: 1,453^rd place)

informatics.indiana.edu

Menczer, F. (1997). ARACHNID: Adaptive Retrieval Agents Choosing Heuristic Neighborhoods for Information Discovery Archived 21 December 2012 at the Wayback Machine. In D. Fisher, ed., Machine Learning: Proceedings of the 14th International Conference (ICML97). Morgan Kaufmann
Menczer, F. and Belew, R.K. (1998). Adaptive Information Agents in Distributed Textual Environments Archived 21 December 2012 at the Wayback Machine. In K. Sycara and M. Wooldridge (eds.) Proc. 2nd Intl. Conf. on Autonomous Agents (Agents '98). ACM Press

mccurley.org (Global: low place; English: low place)

Dill, S.; Kumar, R.; Mccurley, K. S.; Rajagopalan, S.; Sivakumar, D.; Tomkins, A. (2002). "Self-similarity in the web" (PDF). ACM Transactions on Internet Technology. 2 (3): 205–223. doi:10.1145/572326.572328. S2CID 6416041.

mendeley.com (Global: low place; English: low place)

Shestakov, Denis; Bhowmick, Sourav S.; Lim, Ee-Peng (2005). "DEQUE: Querying the Deep Web" (PDF). Data & Knowledge Engineering. 52 (3): 273–311. doi:10.1016/s0169-023x(04)00107-7.^{[permanent dead link]}

nih.gov (Global: 5^th place; English: 5^th place)

pubmed.ncbi.nlm.nih.gov

Lawrence, Steve; C. Lee Giles (8 July 1999). "Accessibility of information on the web". Nature. 400 (6740): 107–9. Bibcode:1999Natur.400..107L. doi:10.1038/21987. PMID 10428673. S2CID 4347646.

nyu.edu (Global: 1,362^nd place; English: 822^nd place)

pages.stern.nyu.edu

Ipeirotis, P., Ntoulas, A., Cho, J., Gravano, L. (2005) Modeling and managing content changes in text databases Archived 5 September 2005 at the Wayback Machine. In Proceedings of the 21st IEEE International Conference on Data Engineering, pages 606-617, April 2005, Tokyo.

poly.edu (Global: low place; English: low place)

cis.poly.edu

Shkapenyuk, V. and Suel, T. (2002). Design and implementation of a high performance distributed web crawler Archived 1 October 2005 at the Wayback Machine. In Proceedings of the 18th International Conference on Data Engineering (ICDE), pages 357-368, San Jose, California. IEEE CS Press.

psu.edu (Global: 299^th place; English: 159^th place)

clgiles.ist.psu.edu

Diligenti, M., Coetzee, F., Lawrence, S., Giles, C. L., and Gori, M. (2000). Focused crawling using context graphs. In Proceedings of 26th International Conference on Very Large Databases (VLDB), pages 527-534, Cairo, Egypt.

researchgate.net (Global: 96^th place; English: 89^th place)

Shervin Daneshpajouh, Mojtaba Mohammadi Nasiri, Mohammad Ghodsi, A Fast Community Based Algorithm for Generating Crawler Seeds Set. In: Proceedings of 4th International Conference on Web Information Systems and Technologies (Webist-2008), Funchal, Portugal, May 2008.
Dong, Hai; Hussain, Farookh Khadeer; Chang, Elizabeth (2009). "State of the Art in Semantic Focused Crawlers". Computational Science and Its Applications – ICCSA 2009. Lecture Notes in Computer Science. Vol. 5593. pp. 910–924. doi:10.1007/978-3-642-02457-3_74. hdl:20.500.11937/48288. ISBN 978-3-642-02456-6.
Dong, Hai; Hussain, Farookh Khadeer (2013). "SOF: A semi-supervised ontology-learning-based focused crawler". Concurrency and Computation: Practice and Experience. 25 (12): 1755–1770. doi:10.1002/cpe.2980. S2CID 205690364.

robotstxt.org (Global: low place; English: low place)

Koster, M. (1996). A standard for robot exclusion Archived 7 November 2007 at the Wayback Machine.
Koster, M. (1993). Guidelines for robots writers Archived 22 April 2005 at the Wayback Machine.

semanticscholar.org (Global: 15^th place; English: 8^th place)

api.semanticscholar.org

Kobayashi, M. & Takeda, K. (2000). "Information retrieval on the web". ACM Computing Surveys. 32 (2): 144–173. doi:10.1145/358923.358934. S2CID 3710903.
Edwards, J.; McCurley, K. S.; and Tomlin, J. A. (2001). "An adaptive model for optimizing performance of an incremental web crawler". Proceedings of the 10th international conference on World Wide Web. pp. 106–113. doi:10.1145/371920.371960. ISBN 978-1581133486. S2CID 10316730. Archived from the original on 25 June 2014. Retrieved 25 January 2007.{{cite book}}: CS1 maint: multiple names: authors list (link)
Lawrence, Steve; C. Lee Giles (8 July 1999). "Accessibility of information on the web". Nature. 400 (6740): 107–9. Bibcode:1999Natur.400..107L. doi:10.1038/21987. PMID 10428673. S2CID 4347646.
Boldi, Paolo; Bruno Codenotti; Massimo Santini; Sebastiano Vigna (2004). "UbiCrawler: a scalable fully distributed Web crawler" (PDF). Software: Practice and Experience. 34 (8): 711–726. doi:10.1002/spe.587. S2CID 325714. Archived from the original (PDF) on 20 March 2009. Retrieved 23 March 2009.
Wu, Jian; Teregowda, Pradeep; Khabsa, Madian; Carman, Stephen; Jordan, Douglas; San Pedro Wandelmer, Jose; Lu, Xin; Mitra, Prasenjit; Giles, C. Lee (2012). "Web crawler middleware for search engine digital libraries". Proceedings of the twelfth international workshop on Web information and data management - WIDM '12. p. 57. doi:10.1145/2389936.2389949. ISBN 9781450317207. S2CID 18513666.
Wu, Jian; Teregowda, Pradeep; Ramírez, Juan Pablo Fernández; Mitra, Prasenjit; Zheng, Shuyi; Giles, C. Lee (2012). "The evolution of a crawling strategy for an academic document search engine". Proceedings of the 3rd Annual ACM Web Science Conference on - Web Sci '12. pp. 340–343. doi:10.1145/2380718.2380762. ISBN 9781450312288. S2CID 16718130.
Dong, Hai; Hussain, Farookh Khadeer (2013). "SOF: A semi-supervised ontology-learning-based focused crawler". Concurrency and Computation: Practice and Experience. 25 (12): 1755–1770. doi:10.1002/cpe.2980. S2CID 205690364.
Cho, Junghoo; Garcia-Molina, Hector (2003). "Effective page refresh policies for Web crawlers". ACM Transactions on Database Systems. 28 (4): 390–426. doi:10.1145/958942.958945. S2CID 147958.
Junghoo Cho; Hector Garcia-Molina (2003). "Estimating frequency of change". ACM Transactions on Internet Technology. 3 (3): 256–290. doi:10.1145/857166.857170. S2CID 9362566.
Dill, S.; Kumar, R.; Mccurley, K. S.; Rajagopalan, S.; Sivakumar, D.; Tomkins, A. (2002). "Self-similarity in the web" (PDF). ACM Transactions on Internet Technology. 2 (3): 205–223. doi:10.1145/572326.572328. S2CID 6416041.
Brin, Sergey; Page, Lawrence (1998). "The anatomy of a large-scale hypertextual Web search engine". Computer Networks and ISDN Systems. 30 (1–7): 107–117. doi:10.1016/s0169-7552(98)00110-x. S2CID 7587743.

semanticweb.com (Global: low place; English: low place)

ITA Labs "ITA Labs Acquisition" Archived 18 March 2014 at the Wayback Machine 20 April 2011 1:28 AM

springer.com (Global: 182^nd place; English: 172^nd place)

Masanès, Julien (15 February 2007). Web Archiving. Springer. p. 1. ISBN 978-3-54046332-0. Retrieved 24 April 2014.

stanford.edu (Global: 194^th place; English: 182^nd place)

ilpubs.stanford.edu

Cho, J.; Garcia-Molina, H.; Page, L. (April 1998). "Calorimetry in High Energy Physics". Seventh International World-Wide Web Conference. Brisbane, Australia. doi:10.1142/3725. ISBN 978-981-02-3400-3. Retrieved 23 March 2009.{{cite journal}}: CS1 maint: periodical has ISBN (link)

infolab.stanford.edu

Brin, Sergey; Page, Lawrence (1998). "The anatomy of a large-scale hypertextual Web search engine". Computer Networks and ISDN Systems. 30 (1–7): 107–117. doi:10.1016/s0169-7552(98)00110-x. S2CID 7587743.

support.google.com (Global: 2,836^th place; English: 2,952^nd place)

"AJAX crawling: Guide for webmasters and developers". Archived from the original on 6 March 2013. Retrieved 17 March 2013.

ucla.edu (Global: 885^th place; English: 584^th place)

oak.cs.ucla.edu

Cho, Junghoo, "Crawling the Web: Discovery and Maintenance of a Large-Scale Web Data" Archived 21 December 2012 at the Wayback Machine, PhD dissertation, Department of Computer Science, Stanford University, November 2001.

uiowa.edu (Global: 2,935^th place; English: 1,725^th place)

dollar.biz.uiowa.edu

Pant, Gautam; Srinivasan, Padmini; Menczer, Filippo (2004). "Crawling the Web" (PDF). In Levene, Mark; Poulovassilis, Alexandra (eds.). Web Dynamics: Adapting to Change in Content, Size, Topology and Use. Springer. pp. 153–178. ISBN 978-3-540-40676-1. Archived from the original (PDF) on 20 March 2009. Retrieved 9 May 2006.

uiuc.edu (Global: 4,225^th place; English: 3,432^nd place)

archive.ncsa.uiuc.edu

Spetka, Scott. "The TkWWW Robot: Beyond Browsing". NCSA. Archived from the original on 3 September 2004. Retrieved 21 November 2010.

unimi.it (Global: 9,149^th place; English: 9,365^th place)

vigna.dsi.unimi.it

Boldi, Paolo; Bruno Codenotti; Massimo Santini; Sebastiano Vigna (2004). "UbiCrawler: a scalable fully distributed Web crawler" (PDF). Software: Practice and Experience. 34 (8): 711–726. doi:10.1002/spe.587. S2CID 325714. Archived from the original (PDF) on 20 March 2009. Retrieved 23 March 2009.
Boldi, Paolo; Massimo Santini; Sebastiano Vigna (2004). "Do Your Worst to Make the Best: Paradoxical Effects in PageRank Incremental Computations" (PDF). Algorithms and Models for the Web-Graph. Lecture Notes in Computer Science. Vol. 3243. pp. 168–180. doi:10.1007/978-3-540-30216-2_14. ISBN 978-3-540-23427-2. Archived from the original (PDF) on 1 October 2005. Retrieved 23 March 2009.

web.archive.org (Global: 1^st place; English: 1^st place)

"Web Crawlers: Browsing the Web". Archived from the original on 6 December 2021.
Spetka, Scott. "The TkWWW Robot: Beyond Browsing". NCSA. Archived from the original on 3 September 2004. Retrieved 21 November 2010.
See definition of scutter on FOAF Project's wiki Archived 13 December 2009 at the Wayback Machine
Edwards, J.; McCurley, K. S.; and Tomlin, J. A. (2001). "An adaptive model for optimizing performance of an incremental web crawler". Proceedings of the 10th international conference on World Wide Web. pp. 106–113. doi:10.1145/371920.371960. ISBN 978-1581133486. S2CID 10316730. Archived from the original on 25 June 2014. Retrieved 25 January 2007.{{cite book}}: CS1 maint: multiple names: authors list (link)
Cho, Junghoo, "Crawling the Web: Discovery and Maintenance of a Large-Scale Web Data" Archived 21 December 2012 at the Wayback Machine, PhD dissertation, Department of Computer Science, Stanford University, November 2001.
Najork, Marc and Janet L. Wiener. "Breadth-first crawling yields high-quality pages". Archived 24 December 2017 at the Wayback Machine In: Proceedings of the Tenth Conference on World Wide Web, pages 114–118, Hong Kong, May 2001. Elsevier Science.
Abiteboul, Serge; Mihai Preda; Gregory Cobena (2003). "Adaptive on-line page importance computation". Proceedings of the 12th international conference on World Wide Web. Budapest, Hungary: ACM. pp. 280–290. doi:10.1145/775152.775192. ISBN 1-58113-680-3. Archived from the original on 20 March 2008. Retrieved 22 March 2009.
Boldi, Paolo; Bruno Codenotti; Massimo Santini; Sebastiano Vigna (2004). "UbiCrawler: a scalable fully distributed Web crawler" (PDF). Software: Practice and Experience. 34 (8): 711–726. doi:10.1002/spe.587. S2CID 325714. Archived from the original (PDF) on 20 March 2009. Retrieved 23 March 2009.
Boldi, Paolo; Massimo Santini; Sebastiano Vigna (2004). "Do Your Worst to Make the Best: Paradoxical Effects in PageRank Incremental Computations" (PDF). Algorithms and Models for the Web-Graph. Lecture Notes in Computer Science. Vol. 3243. pp. 168–180. doi:10.1007/978-3-540-30216-2_14. ISBN 978-3-540-23427-2. Archived from the original (PDF) on 1 October 2005. Retrieved 23 March 2009.
Pant, Gautam; Srinivasan, Padmini; Menczer, Filippo (2004). "Crawling the Web" (PDF). In Levene, Mark; Poulovassilis, Alexandra (eds.). Web Dynamics: Adapting to Change in Content, Size, Topology and Use. Springer. pp. 153–178. ISBN 978-3-540-40676-1. Archived from the original (PDF) on 20 March 2009. Retrieved 9 May 2006.
Cothey, Viv (2004). "Web-crawling reliability" (PDF). Journal of the American Society for Information Science and Technology. 55 (14): 1228–1238. doi:10.1002/asi.20078. Archived from the original (PDF) on 22 September 2017. Retrieved 26 October 2017.
Menczer, F. (1997). ARACHNID: Adaptive Retrieval Agents Choosing Heuristic Neighborhoods for Information Discovery Archived 21 December 2012 at the Wayback Machine. In D. Fisher, ed., Machine Learning: Proceedings of the 14th International Conference (ICML97). Morgan Kaufmann
Menczer, F. and Belew, R.K. (1998). Adaptive Information Agents in Distributed Textual Environments Archived 21 December 2012 at the Wayback Machine. In K. Sycara and M. Wooldridge (eds.) Proc. 2nd Intl. Conf. on Autonomous Agents (Agents '98). ACM Press
Chakrabarti, Soumen; Van Den Berg, Martin; Dom, Byron (1999). "Focused crawling: A new approach to topic-specific Web resource discovery" (PDF). Computer Networks. 31 (11–16): 1623–1640. doi:10.1016/s1389-1286(99)00052-3. Archived from the original (PDF) on 17 March 2004.
Pinkerton, B. (1994). Finding what people want: Experiences with the WebCrawler. In Proceedings of the First World Wide Web Conference, Geneva, Switzerland.
Ipeirotis, P., Ntoulas, A., Cho, J., Gravano, L. (2005) Modeling and managing content changes in text databases Archived 5 September 2005 at the Wayback Machine. In Proceedings of the 21st IEEE International Conference on Data Engineering, pages 606-617, April 2005, Tokyo.
Koster, M. (1996). A standard for robot exclusion Archived 7 November 2007 at the Wayback Machine.
Koster, M. (1993). Guidelines for robots writers Archived 22 April 2005 at the Wayback Machine.
Heydon, Allan; Najork, Marc (26 June 1999). "Mercator: A Scalable, Extensible Web Crawler" (PDF). Archived from the original (PDF) on 19 February 2006. Retrieved 22 March 2009. {{cite journal}}: Cite journal requires |journal= (help)
M. Thelwall; D. Stuart (2006). "Web crawling ethics revisited: Cost, privacy and denial of service". Journal of the American Society for Information Science and Technology. 57 (13): 1771–1779. doi:10.1002/asi.20388. Archived from the original on 7 March 2008. Retrieved 28 January 2008.
Shkapenyuk, V. and Suel, T. (2002). Design and implementation of a high performance distributed web crawler Archived 1 October 2005 at the Wayback Machine. In Proceedings of the 18th International Conference on Data Engineering (ICDE), pages 357-368, San Jose, California. IEEE CS Press.
Shestakov, Denis (2008). Search Interfaces on the Web: Querying and Characterizing Archived 6 July 2014 at the Wayback Machine. TUCS Doctoral Dissertations 104, University of Turku
"AJAX crawling: Guide for webmasters and developers". Archived from the original on 6 March 2013. Retrieved 17 March 2013.
ITA Labs "ITA Labs Acquisition" Archived 18 March 2014 at the Wayback Machine 20 April 2011 1:28 AM
Norton, Quinn (25 January 2007). "Tax takers send in the spiders". Business. Wired. Archived from the original on 22 December 2016. Retrieved 13 October 2017.
"Xenon web crawling initiative: privacy impact assessment (PIA) summary". Ottawa: Government of Canada. 11 April 2017. Archived from the original on 25 September 2017. Retrieved 13 October 2017.

webbrowsersintroduction.com (Global: low place; English: low place)

"Web Crawlers: Browsing the Web". Archived from the original on 6 December 2021.

wired.com (Global: 254^th place; English: 178^th place)

Norton, Quinn (25 January 2007). "Tax takers send in the spiders". Business. Wired. Archived from the original on 22 December 2016. Retrieved 13 October 2017.

wlv.ac.uk (Global: low place; English: low place)

scit.wlv.ac.uk

Cothey, Viv (2004). "Web-crawling reliability" (PDF). Journal of the American Society for Information Science and Technology. 55 (14): 1228–1238. doi:10.1002/asi.20078. Archived from the original (PDF) on 22 September 2017. Retrieved 26 October 2017.
M. Thelwall; D. Stuart (2006). "Web crawling ethics revisited: Cost, privacy and denial of service". Journal of the American Society for Information Science and Technology. 57 (13): 1771–1779. doi:10.1002/asi.20388. Archived from the original on 7 March 2008. Retrieved 28 January 2008.

www10.org (Global: low place; English: low place)

Edwards, J.; McCurley, K. S.; and Tomlin, J. A. (2001). "An adaptive model for optimizing performance of an incremental web crawler". Proceedings of the 10th international conference on World Wide Web. pp. 106–113. doi:10.1145/371920.371960. ISBN 978-1581133486. S2CID 10316730. Archived from the original on 25 June 2014. Retrieved 25 January 2007.{{cite book}}: CS1 maint: multiple names: authors list (link)
Najork, Marc and Janet L. Wiener. "Breadth-first crawling yields high-quality pages". Archived 24 December 2017 at the Wayback Machine In: Proceedings of the Tenth Conference on World Wide Web, pages 114–118, Hong Kong, May 2001. Elsevier Science.

www2003.org (Global: low place; English: low place)

Abiteboul, Serge; Mihai Preda; Gregory Cobena (2003). "Adaptive on-line page importance computation". Proceedings of the 12th international conference on World Wide Web. Budapest, Hungary: ACM. pp. 280–290. doi:10.1145/775152.775192. ISBN 1-58113-680-3. Archived from the original on 20 March 2008. Retrieved 22 March 2009.

Web crawler (English Wikipedia)

adsabs.harvard.edu (Global: 14th place; English: 9th place)

ui.adsabs.harvard.edu

apple.com (Global: 70th place; English: 55th place)

support.apple.com

arxiv.org (Global: 49th place; English: 31st place)

brown.edu (Global: 2,455th place; English: 1,451st place)

cs.brown.edu

canada.ca (Global: 1,114th place; English: 605th place)

chato.cl (Global: low place; English: low place)

csic.es (Global: 2,036th place; English: 2,704th place)

cindoc.csic.es

developers.google.com (Global: 3,679th place; English: 7,859th place)

doi.org (Global: 2nd place; English: 2nd place)

doria.fi (Global: 5,349th place; English: low place)

oa.doria.fi

foaf-project.org (Global: low place; English: low place)

wiki.foaf-project.org

fxpal.com (Global: low place; English: low place)

hdl.handle.net (Global: 66th place; English: 34th place)

indiana.edu (Global: 2,374th place; English: 1,453rd place)

informatics.indiana.edu

mccurley.org (Global: low place; English: low place)

mendeley.com (Global: low place; English: low place)

nih.gov (Global: 5th place; English: 5th place)

pubmed.ncbi.nlm.nih.gov

nyu.edu (Global: 1,362nd place; English: 822nd place)

pages.stern.nyu.edu

poly.edu (Global: low place; English: low place)

cis.poly.edu

psu.edu (Global: 299th place; English: 159th place)

clgiles.ist.psu.edu

researchgate.net (Global: 96th place; English: 89th place)

robotstxt.org (Global: low place; English: low place)

semanticscholar.org (Global: 15th place; English: 8th place)

api.semanticscholar.org

semanticweb.com (Global: low place; English: low place)

springer.com (Global: 182nd place; English: 172nd place)

stanford.edu (Global: 194th place; English: 182nd place)

ilpubs.stanford.edu

infolab.stanford.edu

support.google.com (Global: 2,836th place; English: 2,952nd place)

ucla.edu (Global: 885th place; English: 584th place)

oak.cs.ucla.edu

uiowa.edu (Global: 2,935th place; English: 1,725th place)

dollar.biz.uiowa.edu

uiuc.edu (Global: 4,225th place; English: 3,432nd place)

archive.ncsa.uiuc.edu

unimi.it (Global: 9,149th place; English: 9,365th place)

vigna.dsi.unimi.it

web.archive.org (Global: 1st place; English: 1st place)

webbrowsersintroduction.com (Global: low place; English: low place)

wired.com (Global: 254th place; English: 178th place)

wlv.ac.uk (Global: low place; English: low place)

scit.wlv.ac.uk

www10.org (Global: low place; English: low place)

www2003.org (Global: low place; English: low place)

adsabs.harvard.edu (Global: 14^th place; English: 9^th place)

apple.com (Global: 70^th place; English: 55^th place)

arxiv.org (Global: 49^th place; English: 31^st place)

brown.edu (Global: 2,455^th place; English: 1,451^st place)

canada.ca (Global: 1,114^th place; English: 605^th place)

csic.es (Global: 2,036^th place; English: 2,704^th place)

developers.google.com (Global: 3,679^th place; English: 7,859^th place)

doi.org (Global: 2^nd place; English: 2^nd place)

doria.fi (Global: 5,349^th place; English: low place)

hdl.handle.net (Global: 66^th place; English: 34^th place)

indiana.edu (Global: 2,374^th place; English: 1,453^rd place)

nih.gov (Global: 5^th place; English: 5^th place)

nyu.edu (Global: 1,362^nd place; English: 822^nd place)

psu.edu (Global: 299^th place; English: 159^th place)

researchgate.net (Global: 96^th place; English: 89^th place)

semanticscholar.org (Global: 15^th place; English: 8^th place)

springer.com (Global: 182^nd place; English: 172^nd place)

stanford.edu (Global: 194^th place; English: 182^nd place)

support.google.com (Global: 2,836^th place; English: 2,952^nd place)

ucla.edu (Global: 885^th place; English: 584^th place)

uiowa.edu (Global: 2,935^th place; English: 1,725^th place)

uiuc.edu (Global: 4,225^th place; English: 3,432^nd place)

unimi.it (Global: 9,149^th place; English: 9,365^th place)

web.archive.org (Global: 1^st place; English: 1^st place)

wired.com (Global: 254^th place; English: 178^th place)