robots.txt

Maljonic

Can't get enough of FH
Joined
Dec 22, 2003
Messages
1,297
robots.text

I'm sure this is a very old question but what exactly do I do with robots text? Do I need to do anything at all? Can I just not bother with it?
 

sibanac

Fledgling Freddie
Joined
Dec 19, 2003
Messages
824
Maljonic said:
I'm sure this is a very old question but what exactly do I do with robots text? Do I need to do anything at all? Can I just not bother with it?

It is used to tell webcrawlers (indexing from search engines) what they can and cant index from your page.
this can be a good option for /cgi-bin or other dynamic fast changing pages.


look here for more info
 

Jonty

Fledgling Freddie
Joined
Dec 22, 2003
Messages
1,411
Hi Maljonic

Sibanac is spot on in that robots.txt effectively helps search engines decide which pages they should crawl (protecting private directories is also a major use of this technology).

More recently, you can also use a meta tag to save you have to worry about creating robots.txt files; e.g.


Code:
<meta content="all" name="robots" />
If you search Google or the likes for this method you'll find all the information you need. Note, however, that not all search engines support this meta robots format (although most major operators do I believe) so the traditional robots.txt method may prove to be the most effective.

Kind Regards
 

Maljonic

Can't get enough of FH
Joined
Dec 22, 2003
Messages
1,297
Thanks Jonty, I don't have any private directories as yet and I am using the meta tag you mentioned. And sibanac, I just read that stuff before I posted this; I'm just wondering if it's okay not to have robots text? Or if it is better to have it, how do I create it and where do I put it?
 

Maljonic

Can't get enough of FH
Joined
Dec 22, 2003
Messages
1,297
do I write it as a web page then or is it just a text file, like you might write in notepad or something?
 

sibanac

Fledgling Freddie
Joined
Dec 19, 2003
Messages
824
Maljonic said:
do I write it as a web page then or is it just a text file, like you might write in notepad or something?

just a notepad thing will do
Code:
# My robot.txt file
# <- means comment 
# this robot.txt file allows all robots to index the site except /images/* and /cgi-bin/*

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
 

Mike

Fledgling Freddie
Joined
Dec 11, 2003
Messages
27
Check out http://www.robotstxt.org for info on robots.txt, how it works, etc. It doesn't look particularly appealing, but all the info you need is there.

Be aware that the file has to be readable, by anyone. So by adding a private dir to it, you are effectively telling everyone the url of that private dir :) Therefore you should password such dirs, if you add them to robots.txt
 

Users Who Are Viewing This Thread (Users: 0, Guests: 1)

Top Bottom