Comments on: Advances in Crawling the Web http://www.semclubhouse.com/advances-in-crawling-the-web/ Sat, 11 Oct 2008 21:30:51 +0000 http://wordpress.org/?v=2.5.1 By: Links of the Week March 7th, 2008 | .eduGuru http://www.semclubhouse.com/advances-in-crawling-the-web/#comment-64 Links of the Week March 7th, 2008 | .eduGuru Fri, 07 Mar 2008 16:38:40 +0000 http://www.semclubhouse.com/advances-in-crawling-the-web.html#comment-64 [...] Advances in Crawling the Web - There are 3 major parts to what a search engine does. What are some of the problems facing crawling programs? [...] [...] Advances in Crawling the Web - There are 3 major parts to what a search engine does. What are some of the problems facing crawling programs? [...]

]]> By: Bill http://www.semclubhouse.com/advances-in-crawling-the-web/#comment-63 Bill Tue, 04 Mar 2008 15:13:31 +0000 http://www.semclubhouse.com/advances-in-crawling-the-web.html#comment-63 You're welcome, James. Funny, taking down your own site. Guess it was a good thing that it wasn't someone elses. Thanks for sharing that story. I like the old usenet posts which talk about the early days of crawlers, and people trying to figure out why others were spending so much time grabbing pages from their sites. There are a lot of elements to what makes a search engine work that go on behind the scenes, so it's great when someone provides as much depth and detail as the Texas A&M researchers did in their paper. You’re welcome, James.

Funny, taking down your own site. Guess it was a good thing that it wasn’t someone elses. Thanks for sharing that story. I like the old usenet posts which talk about the early days of crawlers, and people trying to figure out why others were spending so much time grabbing pages from their sites.

There are a lot of elements to what makes a search engine work that go on behind the scenes, so it’s great when someone provides as much depth and detail as the Texas A&M researchers did in their paper.

]]> By: James http://www.semclubhouse.com/advances-in-crawling-the-web/#comment-62 James Mon, 03 Mar 2008 14:12:59 +0000 http://www.semclubhouse.com/advances-in-crawling-the-web.html#comment-62 I built myself a simple crawler and learnt most of this along the way... The problems you com up against are really interesting. I like the politeness one....for my first attempt I took down my own site by using multi-curl....ha ha ahh the lessons we learn. Nice article, thanks.... I built myself a simple crawler and learnt most of this along the way… The problems you com up against are really interesting.
I like the politeness one….for my first attempt I took down my own site by using multi-curl….ha ha ahh the lessons we learn.

Nice article, thanks….

]]>