Crawlers

[MKJ]

Can someone explain about crawlers!

My site has been inundated with crawlers recently. Is this a good sign? Do these crawlers appear on every site? Why are they continually scanning the site? Googlebot is all over the site at the moment. How do these crawlers really work and what are they looking for?

 

[Dommers]

Wasn't this explained to you in a previous thread? If not they browse your sites for content so they can display them when someones searches for something. Say your site is about the breeding habits of kit kats, google would gather this information so that when someone typed in the breeding habits of kit kats your site would come up. They also save you website so if your site goes offline they have a copy of in a cache

 

[MKJ]

Well why do they keep crawling over and over? Ah bugger it just making a nuisance of myself. Whisky you know. You know what I dislike? - order. Chaos is better for me. Youse lot have ordered minds whereas mine is totally different. I reckon I would make a shit-hot hacker. Youse lot are too stuck in the mud. Who would win in the battle for supremacy in the website designing or hacking world? You or me? Me for sure .

 

[Dommers]

How would having a chaotic mind make you a better hacker. I would have thought you needed a cool and clear head to hack..

 

[Shovel]

Crawlers regularly revisit your site to check for updated content and new links.

 

[MKJ]

My mind is like 'steel' when I want it to be.

Yu ever get drunk?

 

[TdC]

have a look at this, and add a file like it to your webserver's root. you'll be rid of a goodly amount then.

if you spot a new one in your webserver's logs then you could add it potentially and if it behaves (and it should) it will leave again. you'll notice that as soon as a bot hits your site it will look for the robots.txt file and read it. the rest comes naturally.

my site's current top 10 useragent hits are made up thusly:

1 1452 29.85% Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.12) Ge
2 504 10.36% Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.co
3 471 9.68% Mozilla/5.0 (Windows; U; Windows NT 5.1; nl-NL; rv:1.7.7) Gec
4 323 6.64% Mozilla/5.0 (Windows; U; Windows NT 5.1; nl-NL; rv:1.7.12) Ge
5 301 6.19% MJ12bot/v1.0.3 (http://majestic12.co.uk/bot.php?+)
6 278 5.71% msnbot/1.0 (+http://search.msn.com/msnbot.htm)
7 273 5.61% Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET
8 158 3.25% MJ12bot/v1.0.2 (http://majestic12.co.uk/bot.php?+)
9 137 2.82% Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/
10 120 2.47% MJ12bot/v1.0.4 (http://majestic12.co.uk/bot.php?+)

of these, 6 are bots. it's a balance you have to find. it's nice to exist in search engines, but they do tend to eat bandwidth.

 

[MYstIC G]

which is only a problem if you've got a small willy

 

[TdC]

hey, it's a whole MBit

 

[MKJ]

That is the last time I pick up a bottle of whisky - I hope. I always go absolutely mental onnit. Wonder I didn't start trying to pick a big argument in this section of Freddy's. Posted utter nonsense in a number of others. At least I never wrecked my site like I normally do .

Stuck a new site on today - looks like it is all to do with smoking pot? How can that be legal? Legitimate site as far as I can tell too. Not going to put a link on my front page to it but it is in my 'Food & Drink' section. Lot less damaging that booze I think though I don't smoke. Did try smoking stuff at one time with a mate. I didn't know what the hell I was doing and when I took a puff once from the wrong end I turned green. That was enough for me. Am gonna stick to lager from now on.

My site is coming on leaps and bounds now though.