Daily Spidering

2004 March 15 at 21:45 » Tagged as :

A well designed spider will hit urls that are regularly updated more often than those URLs that are infrequently updated.

In recent weeks I have noticed that google seems to be spidering some parts of my website on a daily basis. These pages have not been updated and I do not send a modified by header. Anyway most intelligent spiders (out of which google's out to be the top of the class) calculate a message digest to find out if a page has really been modified.

What's really puzzling is that pages that have been modified more than two weeks ago haven't been reindexed yet.

Anyway why this obsession with google? The answer is that though goole may not have a monopoly in the search business it is the engine that almost every techy seems to use. And that's the kind of people who visit my site.