Jump to content

spiders


taylrl

Recommended Posts

I was just looking at the home page, and i noticed that most of the people on this website are spiders. i know what a spider is, but that is about as far as my knowledge goes. Are they deployed automatically? Do they move? if so how do they decide where to stop, and when to go? A bit more info would be appreciated.

Link to comment
Share on other sites

I was just looking at the home page, and i noticed that most of the people on this website are spiders. i know what a spider is, but that is about as far as my knowledge goes. Are they deployed automatically? Do they move? if so how do they decide where to stop, and when to go? A bit more info would be appreciated.

 

Its actually quite interesting!

 

Depending on their job these can be called trawlers too.

 

The way most work is simple (The example below is how Google's bots work)

 

1) Start on a page scan the page's source for links. These are then added to a temp table for later use. (Google also downloads caches images and some file types for easy loading).

 

2) Grab the links to any media on the page and add them to a record archive. If these already exist then the count is incrimented.

 

3) Next load the temporary table and then repeat the process for each of the found links.

 

I left out about googles rating system because it makes things far more complex.

 

Do they move as I explained above they follow links and by doing so move out in an ever widening grid.

Do they stop this is a bit harder. If they originbate from a small company then they are limited by how far they can spread the net because this is quite memory intensive, for something like google the net size is all but limitless. So they only stop by memory limitations :)

 

Hope I helped explain it a bit better if you have any questions I'll do my best to explain what I wrote above... hope it makes sence :D

 

Cheers,

 

Ryan Jones

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.